python 3.x - How to handle min_itemsize exception in writing to pandas HDFStore -
i using pandas hdfstore store dfs have created data.
store = pd.hdfstore(storename, ...) file in downloaded_files:     try:         gzip.open(file) f:             data = json.loads(f.read())             df = json_normalize(data)                store.append(storekey, df, format='table', append=true)     except typeerror:         pass         #file error i have received error:
valueerror: trying store string len [82] in [values_block_2] column column has limit of [72]! consider using min_itemsize preset sizes on these columns i found possible set min_itemsize column involved not viable solution not know max length encounter , columns encounter problem.
is there solution automatically catch exception , handle each item occur?
i think can way:
store.append(storekey, df, format='table', append=true, min_itemsize={'long_string_column': 200}) basically it's similar following create table sql statement:
create table df(   id     int,   str    varchar(200) ); where 200 maximal allowed length str column
the following links might helpful:
hdfstore.append(string, dataframe) fails when string column contents longer there
pandas pytable: how specify min_itemsize of elements of multiindex
Comments
Post a Comment