^ i=QbddlmZddlmZmZddlZddlmZmZm Z m Z ddl Z ddl mZddlmZddlmZddlZddlmZmZerdd lmZdd lmZmZd;dZe ddZ d?d@d!ZdAd'ZdBd(Z dCdDd*Z dEd.Z!ed/ dFdGd:Z"dS)H) annotations)abc defaultdictN) TYPE_CHECKINGAny DefaultDictoverloadconvert_json_to_lines) set_module) is_scalar) DataFrameSeries)Iterable) IgnoreRaiseScalarsstrreturnch|ddks|ddkr|S|dd}t|S)zJ Helper function that converts JSON lists to line delimited JSON. r[]r )rs D/usr/local/lib/python3.11/dist-packages/pandas/io/json/_normalize.pyconvert_to_line_delimitsr's= Q43;;1R5C<< !B$A  # ##.dsdictprefixseplevelint max_level int | Nonedict[str, Any]cdSNrr r!r"r$s rnested_to_recordr+4s Sr list[dict]list[dict[str, Any]]cdSr(r)r*s rr+r+>s 3r.dict | list[dict]%dict[str, Any] | list[dict[str, Any]]c Hd}t|tr|g}d}g}|D]}tj|}|D]\} } t| t st | } |dkr| } n||z| z} t| tr|'||kr!|dkr|| } | || <x|| } |t| | ||dz|| ||r|dS|S)a A simplified json_normalize Converts a nested dict into a flat dict ("record"), unlike json_normalize, it does not attempt to extract a subset of the data. Parameters ---------- ds : dict or list of dicts prefix: the prefix, optional, default: "" sep : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar level: int, optional, default: 0 The number of levels in the json string. max_level: int, optional, default: None The max depth to normalize. Returns ------- d - dict or list of dicts, matching `ds` Examples -------- >>> nested_to_record( ... dict(flat1=1, dict1=dict(c=1, d=2), nested=dict(e=dict(c=1, d=2), d=2)) ... ) {'flat1': 1, 'dict1.c': 1, 'dict1.d': 2, 'nested.e.c': 1, 'nested.e.d': 2, 'nested.d': 2} FTrNr) isinstancercopydeepcopyitemsrpopupdater+append) rr r!r"r$ singletonnew_dsdnew_dkvnewkeys rr+r+HsPXI"dT F  a  GGII Q QDAqa%% FFzz#) a&& %%9*<*<A:: ! A$%E&M ! A LL)!VS%!)YOO P P P P eay Mrdatar key_stringnormalized_dict separatorct|trM|D]7\}}|||}|s||}t ||||8n|||<|S)a3 Main recursive function Designed for the most basic use case of pd.json_normalize(data) intended as a performance improvement, see #15621 Parameters ---------- data : Any Type dependent on types contained within nested Json key_string : str New key (with separator(s) in) for data normalized_dict : dict The new normalized/flattened Json dict separator : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar rBrCrDrE)r4rr7 removeprefix_normalize_json)rBrCrDrEkeyvaluenew_keys rrIrIs.$+**,,  JC#5Y555G :!..y99 " /#       '+ # rcd|D}td|Ddi|}i||S)aw Order the top level keys and then recursively go to depth Parameters ---------- data : dict or list of dicts separator : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar Returns ------- dict or list of dicts, matching `normalized_json_object` cDi|]\}}t|t||Sr)r4r.0r?r@s r z+_normalize_json_ordered..s-JJJ$!QjD6I6IJAJJJrcDi|]\}}t|t||Sr)rOrPs rrRz+_normalize_json_ordered..s- C C Ctq!z!T/B/B Ca C C Crr/rG)r7rI)rBrE top_dict_ nested_dict_s r_normalize_json_orderedrVshKJ$**,,JJJI" C Ctzz|| C C C L )i (< ((rdict | list[dict] | Anyci}t|trt|}n%t|trfd|D}|S|S)a An optimized basic json_normalize Converts a nested dict into a flat dict ("record"), unlike json_normalize and nested_to_record it doesn't do anything clever. But for the most basic use cases it enhances performance. E.g. pd.json_normalize(data) Parameters ---------- ds : dict or list of dicts sep : str, default '.' Nested records will generate names separated by sep, e.g., for sep='.', { 'foo' : { 'bar' : 0 } } -> foo.bar Returns ------- frame : DataFrame d - dict or list of dicts, matching `normalized_json_object` Examples -------- >>> _simple_json_normalize( ... { ... "flat1": 1, ... "dict1": {"c": 1, "d": 2}, ... "nested": {"e": {"c": 1, "d": 2}, "d": 2}, ... } ... ) {'flat1': 1, 'dict1.c': 1, 'dict1.d': 2, 'nested.e.c': 1, 'nested.e.d': 2, 'nested.d': 2} )rBrEc2g|]}t|S)r!)_simple_json_normalize)rQrowr!s r z*_simple_json_normalize.. s'SSS 6s D D DSSSr)r4rrVlist)rr!normalized_json_objectnormalized_json_lists ` rr[r[smV "d$!8bC!P!P!P B  $SSSSPRSSS## !!rmeta"str | list[str | list[str]] | NoneNonecl|dSt|trdS|D]}t|trB|D]>}t|ts'tdt |jd|?Yt|ts'tdt |jd|dS)aD Validate that meta parameter contains only strings or lists of strings. Parameters ---------- meta : str or list of str or list of list of str or None The meta parameter to validate. Raises ------ TypeError If meta contains elements that are not strings or lists of strings. Nz9All elements in nested meta paths must be strings. Found z: zBAll elements in 'meta' must be strings or lists of strings. Found )r4rr^ TypeErrortype__name__)raitemsubitems r_validate_metarjs  |$   dD ! !   !'3//#G!%g!7GG;BGG  D#&& 9d,990499    rpandasraisedict | list[dict] | Series record_pathstr | list | None meta_prefix str | None record_prefixerrorsrrc t| d"d#fd d$fd t|tr|j}nd }t|tr|st St|t r|g}nt|tjrt|tst |}t|D]i\} } t| t rt| rtj | ri|| <Ddt| j} t!| nt"|(|&|$" t t%||S|Yt'd|Drt)|}t ||} | fd} | St|ts|g}|g}nt|ts|g}d|Dggt-tfdDd%d&f d ||idt } | fd} D]\} }||| z} | | vrt1d| dt3j|t6 }|jd!krCt3jt=|ft6 }t|D] \} }||| < || | <||| _| S)'a> Normalize semi-structured JSON data into a flat table. This method is designed to transform semi-structured JSON data, such as nested dictionaries or lists, into a flat table. This is particularly useful when handling JSON-like data structures that contain deeply nested fields. Parameters ---------- data : dict, list of dicts, or Series of dicts Unserialized JSON objects. record_path : str or list of str, default None Path in each object to list of records. If not passed, data will be assumed to be an array of records. meta : list of paths (str or list of str), default None Fields to use as metadata for each record in resulting table. meta_prefix : str, default None String to prefix records with dotted path, e.g. foo.bar.field if meta is ['foo', 'bar']. record_prefix : str, default None String to prefix records with dotted path, e.g. foo.bar.field if path to records is ['foo', 'bar']. errors : {'raise', 'ignore'}, default 'raise' Configures error handling. * 'ignore' : will ignore KeyError if keys listed in meta are not always present. * 'raise' : will raise KeyError if keys listed in meta are not always present. sep : str, default '.' Nested records will generate names separated by sep. e.g., for sep='.', {'foo': {'bar': 0}} -> foo.bar. max_level : int, default None Max number of levels(depth of dict) to normalize. if None, normalizes all levels. Returns ------- DataFrame The normalized data, represented as a pandas DataFrame. See Also -------- DataFrame : Two-dimensional, size-mutable, potentially heterogeneous tabular data. Series : One-dimensional ndarray with axis labels (including time series). Examples -------- >>> data = [ ... {"id": 1, "name": {"first": "Coleen", "last": "Volk"}}, ... {"name": {"given": "Mark", "family": "Regner"}}, ... {"id": 2, "name": "Faye Raker"}, ... ] >>> pd.json_normalize(data) id name.first name.last name.given name.family name 0 1.0 Coleen Volk NaN NaN NaN 1 NaN NaN NaN Mark Regner NaN 2 2.0 NaN NaN NaN NaN Faye Raker >>> data = [ ... { ... "id": 1, ... "name": "Cole Volk", ... "fitness": {"height": 130, "weight": 60}, ... }, ... {"name": "Mark Reg", "fitness": {"height": 130, "weight": 60}}, ... { ... "id": 2, ... "name": "Faye Raker", ... "fitness": {"height": 130, "weight": 60}, ... }, ... ] >>> pd.json_normalize(data, max_level=0) id name fitness 0 1.0 Cole Volk {'height': 130, 'weight': 60} 1 NaN Mark Reg {'height': 130, 'weight': 60} 2 2.0 Faye Raker {'height': 130, 'weight': 60} Normalizes nested data up to level 1. >>> data = [ ... { ... "id": 1, ... "name": "Cole Volk", ... "fitness": {"height": 130, "weight": 60}, ... }, ... {"name": "Mark Reg", "fitness": {"height": 130, "weight": 60}}, ... { ... "id": 2, ... "name": "Faye Raker", ... "fitness": {"height": 130, "weight": 60}, ... }, ... ] >>> pd.json_normalize(data, max_level=1) id name fitness.height fitness.weight 0 1.0 Cole Volk 130 60 1 NaN Mark Reg 130 60 2 2.0 Faye Raker 130 60 >>> data = [ ... { ... "id": 1, ... "name": "Cole Volk", ... "fitness": {"height": 130, "weight": 60}, ... }, ... {"name": "Mark Reg", "fitness": {"height": 130, "weight": 60}}, ... { ... "id": 2, ... "name": "Faye Raker", ... "fitness": {"height": 130, "weight": 60}, ... }, ... ] >>> series = pd.Series(data, index=pd.Index(["a", "b", "c"])) >>> pd.json_normalize(series) id name fitness.height fitness.weight a 1.0 Cole Volk 130 60 b NaN Mark Reg 130 60 c 2.0 Faye Raker 130 60 >>> data = [ ... { ... "state": "Florida", ... "shortname": "FL", ... "info": {"governor": "Rick Scott"}, ... "counties": [ ... {"name": "Dade", "population": 12345}, ... {"name": "Broward", "population": 40000}, ... {"name": "Palm Beach", "population": 60000}, ... ], ... }, ... { ... "state": "Ohio", ... "shortname": "OH", ... "info": {"governor": "John Kasich"}, ... "counties": [ ... {"name": "Summit", "population": 1234}, ... {"name": "Cuyahoga", "population": 1337}, ... ], ... }, ... ] >>> result = pd.json_normalize( ... data, "counties", ["state", "shortname", ["info", "governor"]] ... ) >>> result name population state shortname info.governor 0 Dade 12345 Florida FL Rick Scott 1 Broward 40000 Florida FL Rick Scott 2 Palm Beach 60000 Florida FL Rick Scott 3 Summit 1234 Ohio OH John Kasich 4 Cuyahoga 1337 Ohio OH John Kasich >>> data = {"A": [1, 2]} >>> pd.json_normalize(data, "A", record_prefix="Prefix.") Prefix.0 0 1 1 2 Returns normalized data with columns prefixed with the given string. Fjsr&spec list | strextract_recordboolrScalar | Iterablec4|} t|tr|D]}|t|||}n||}nV#t$rI}|rtd|d|dkrtjcYd}~Std|d|d|d}~wwxYw|S)zInternal function to pull fieldNzKey zS not found. If specifying a record_path, all elements of data should have the path.ignorez) not found. To replace missing values of z% with np.nan, pass in errors='ignore')r4r^KeyErrornpnan)rurvrxresultfielderss r _pull_fieldz#json_normalize.._pull_fields $%% &!++E~&uoo-#E]FF+      21222!!v 7177q777  s#tj|rg}n't dt |jd||S)z Internal function to pull field for records, and similar to _pull_field, but require to return list. And will raise error if has non iterable value. T)rxz(Path must contain list or null, but got z at )r4r^pdisnullrerfrg)rurvrrs r _pull_recordsz%json_normalize.._pull_recordss Rd;;;&$'' y   C#F||4CC:>CC rNz9All items in data must be of type dict or NA-like, found rZ)indexc3RK|]"}d|DV#dS)c8g|]}t|tSr)rO)rQxs rr]z,json_normalize...5s"888 1d##888rN)values)rQys r z!json_normalize..5s9GGQ88QXXZZ888GGGGGGrr!r$c|Sr(r)rrrs rz json_normalize..@s 5Jq5J5Jr)columnscBg|]}t|tr|n|gSr))r4r^)rQms rr]z"json_normalize..Js- = = =1*Q%% .QQA3 = = =rc:g|]}|Sr))join)rQvalr!s rr]z"json_normalize..Qs#0003#000rrr"r#rcc t|tr|g}t|dkrp|D]k}t dD]0\}}|dzt|kr ||d||<1 ||d|dd||dzldS|D]} ||d}fd|D} t|t dD]S\}}|dzt|kr ||}n |||d}||T|dS)NrT)strictrrr"cbg|]+}t|trt|n|,S)r)r4rr+)rQrr$r!s rr]z>json_normalize.._recursive_extract..`sR"!T**$QC9EEEEr)r4rlenzipr:extend)rBpath seen_metar"objrrJrecsmeta_val_metarr_recursive_extractlengthsr$ meta_keys meta_valsrecordsr!s rrz*json_normalize.._recursive_extractSs dD ! ! 6D t99q== W W #E9T B B BCCHCqyCHH,,)4S#b')B)B #""3tAw<abb9ETUIVVVVV  W W % %$}S$q'22" s4yy))) #E9T B B B44HCqy3s88++#,S>#.;sCK#@#@cN))(3333t$$$$# % %rrc|Sr(r)rs rrz json_normalize..vsM1F11F1FrzConflicting metadata name z, need distinguishing prefix )dtyper)F)rur&rvrwrxryrrz)rur&rvrwrr^)r)r"r#rrc) rjr4rrr^rrrrr enumerater risnarfrgreNotImplementedErrorr[anyr+renamerr7 ValueErrorr~arrayobjectndimemptyrrepeat)rBrnrarprrrsr!r$rirhmsgrr?r@rrrrrrrrrrs ```` @@@@@@@@rjson_normalizer/sLT4FK:($ $"d"{{ D$  "v D#, ' '" 40E0E"Dzz  % %GAt$%%  %274== %Q3!$ZZ033 nn$ %"!  L    !  /#>>>eLLLL GG$GGG G G H$DcYGGGD4u---  $]]+J+J+J+J]KKF  T * *$"m  | d # #v = = = = =EGG(..I0000%000I%%%%%%%%%%%%%%%%<t["A6666 w  F 'F'F'F'FGG!!++1  "aA ;;MQMMM  !6*** ;??Xs1vviv666F#A,,  3q MM'**q  ||G,, Mr)rrrr)....) rrr rr!rr"r#r$r%rr&) rr,r rr!rr"r#r$r%rr-)r/r0rN) rr1r rr!rr"r#r$r%rr2) rBrrCrrDr&rErrr&)rBr&rErrr&)r0)rr1r!rrrW)rarbrrc)NNNNrlr0N)rBrmrnrorarbrprqrrrqrsrr!rr$r%rr)# __future__r collectionsrrr5typingrrrr numpyr~pandas._libs.writersr pandas.util._decoratorsr pandas.core.dtypes.commonr rkrrrcollections.abcrpandas._typingrrrr+rIrVr[rjrr)rrrs#""""" 666666......////// (((((( $ $ $ $       NNNNNb&&&&R))))62"2"2"2"2"j> H&*/3" $! ^^^^^^^r