python - Merge a lot of df using pandas -

i have big df , use 'chunksize' divide it. after that use loop go through interval of df , next loop condition , next want merge of df. try 'concat(df)' return error. method 'join' not convenient because have 400 df. how can concatenate this? code

el = pd.read_csv('df2.csv', iterator=true, chunksize=100000) buys = pd.read_excel('smartphone.xlsx') buys['date'] = pd.to_datetime(buys['date']) dates1 = buys['date'] ids1 = buys['id'] in el: i['used_at'] = pd.to_datetime(i['used_at']) df = i.sort_values(['id', 'used_at']) dates = df['used_at'] ids = df['id'] urls = df['url'] i, (id, date, url, id1, date1) in enumerate(zip(ids, dates, urls, ids1, dates1)):     df1 = df[(df['id'] == ids1[i]) & (df['used_at'] < (dates1[i] + dateutil.relativedelta.relativedelta(days=5)).replace(hour=0, minute=0, second=0)) & (df['used_at'] > (dates1[i] - dateutil.relativedelta.relativedelta(months=1)).replace(day=1, hour=0, minute=0, second=0))]     df1 = dataframe(df1)     if df1.empty:         continue     else:         df_upd = concat(df1, ignore_index=true)         book = load_workbook('report_buy2.xlsx')         writer = pd.excelwriter('report_buy2.xlsx', engine='openpyxl')         writer.book = book         writer.sheets = dict((ws.title, ws) ws in book.worksheets)         df_upd.to_excel(writer, "main")          writer.save() 

the pandas v0.18.1 documentation merge, join, , concatenate states list of dataframes argument / parameter concat function sample:

import pandas pd # ... generate df1 et al. frames = [df1, df2, df3] result = pd.concat(frames) 

in preamble states "the concat function (in main pandas namespace) of heavy lifting of performing concatenation operations along axis while performing optional set logic (union or intersection) of indexes (if any) on other axes. note “if any” because there single possible axis of concatenation series."

so in case, df chunks being dataframes should collected in container , container (above named frames) given concat function return new dataframe chunks concatenated.

if function accepts general iterables might looked there ...