python 3.x - How to handle min_itemsize exception in writing to pandas HDFStore -
i using pandas hdfstore store dfs have created data.
store = pd.hdfstore(storename, ...) file in downloaded_files: try: gzip.open(file) f: data = json.loads(f.read()) df = json_normalize(data) store.append(storekey, df, format='table', append=true) except typeerror: pass #file error
i have received error:
valueerror: trying store string len [82] in [values_block_2] column column has limit of [72]! consider using min_itemsize preset sizes on these columns
i found possible set min_itemsize column involved not viable solution not know max length encounter , columns encounter problem.
is there solution automatically catch exception , handle each item occur?
i think can way:
store.append(storekey, df, format='table', append=true, min_itemsize={'long_string_column': 200})
basically it's similar following create table
sql statement:
create table df( id int, str varchar(200) );
where 200 maximal allowed length str
column
the following links might helpful:
hdfstore.append(string, dataframe) fails when string column contents longer there
pandas pytable: how specify min_itemsize of elements of multiindex
Comments
Post a Comment