Skip to content

fix: increase streaming chunk size for CSV and Excel view downloads#1817

Open
jacalata wants to merge 1 commit into
developmentfrom
jac/fix-large-view-download
Open

fix: increase streaming chunk size for CSV and Excel view downloads#1817
jacalata wants to merge 1 commit into
developmentfrom
jac/fix-large-view-download

Conversation

@jacalata

Copy link
Copy Markdown
Contributor

Summary

  • Raises iter_content chunk size from 1024 to 65536 bytes in _get_view_csv and _get_view_excel
  • The 1 KB default required ~200,000 generator iterations for a 200 MB download, causing apparent hangs when callers materialise the full stream with b''.join(view.csv)

Closes #1374

Test plan

  • Existing test_populate_csv, test_populate_excel, test_filter_excel, test_populate_csv_default_maxage all pass

🤖 Generated with Claude Code

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown

Coverage

Coverage Report
FileStmtsMissCoverMissing
tableauserverclient
   __init__.py50100% 
   config.py150100% 
   datetime_helpers.py2511 96%
   exponential_backoff.py200100% 
   filesys_helpers.py310100% 
   namespace.py2633 88%
tableauserverclient/bin
   __init__.py20100% 
   _version.py358212212 41%
tableauserverclient/helpers
   __init__.py10100% 
   logging.py20100% 
   strings.py3111 97%
tableauserverclient/models
   __init__.py460100% 
   collection_item.py4177 83%
   column_item.py553232 42%
   connection_credentials.py351111 69%
   connection_item.py941414 85%
   custom_view_item.py1442121 85%
   data_acceleration_report_item.py5411 98%
   data_alert_item.py15844 97%
   data_freshness_policy_item.py1551515 90%
   database_item.py2073636 83%
   datasource_item.py3001212 96%
   dqw_item.py10455 95%
   exceptions.py40100% 
   extensions_item.py13244 97%
   extract_item.py4444 91%
   favorites_item.py6988 88%
   fileupload_item.py190100% 
   flow_item.py1491010 93%
   flow_run_item.py710100% 
   group_item.py8966 93%
   groupset_item.py4977 86%
   interval_item.py1823232 82%
   job_item.py1871010 95%
   linked_tasks_item.py7911 99%
   location_item.py2922 93%
   metric_item.py1291313 90%
   oidc_item.py6333 95%
   pagination_item.py3411 97%
   permissions_item.py1111212 89%
   project_item.py2073131 85%
   property_decorators.py1001818 82%
   reference_item.py2622 92%
   revision_item.py5911 98%
   schedule_item.py20966 97%
   server_info_item.py3777 81%
   site_item.py6361313 98%
   subscription_item.py10122 98%
   table_item.py1191818 85%
   tableau_auth.py612525 59%
   tableau_types.py2711 96%
   tag_item.py150100% 
   target.py60100% 
   task_item.py5622 96%
   user_item.py3101818 94%
   view_item.py2201616 93%
   virtual_connection_item.py6488 88%
   webhook_item.py6911 99%
   workbook_item.py3621616 96%
tableauserverclient/server
   __init__.py90100% 
   exceptions.py40100% 
   filter.py2111 95%
   pager.py3311 97%
   query.py1431515 90%
   request_factory.py1335195195 85%
   request_options.py38655 99%
   server.py1882323 88%
   sort.py60100% 
tableauserverclient/server/endpoint
   __init__.py350100% 
   auth_endpoint.py771111 86%
   custom_views_endpoint.py1521212 92%
   data_acceleration_report_endpoint.py210100% 
   data_alert_endpoint.py942323 76%
   databases_endpoint.py1113030 73%
   datasources_endpoint.py3403535 90%
   default_permissions_endpoint.py4433 93%
   dqw_endpoint.py451616 64%
   endpoint.py1871818 90%
   exceptions.py7766 92%
   extensions_endpoint.py310100% 
   favorites_endpoint.py942222 77%
   fileuploads_endpoint.py510100% 
   flow_runs_endpoint.py6299 85%
   flow_task_endpoint.py2122 90%
   flows_endpoint.py2165555 75%
   groups_endpoint.py12699 93%
   groupsets_endpoint.py7277 90%
   jobs_endpoint.py6799 87%
   linked_tasks_endpoint.py370100% 
   metadata_endpoint.py881414 84%
   metrics_endpoint.py5566 89%
   oidc_endpoint.py4211 98%
   permissions_endpoint.py4433 93%
   projects_endpoint.py1572424 85%
   resource_tagger.py1273535 72%
   schedules_endpoint.py1191111 91%
   server_info_endpoint.py361010 72%
   sites_endpoint.py1302727 79%
   subscriptions_endpoint.py561414 75%
   tables_endpoint.py1103636 67%
   tasks_endpoint.py6366 90%
   users_endpoint.py18388 96%
   views_endpoint.py15199 94%
   virtual_connections_endpoint.py1131010 91%
   webhooks_endpoint.py5499 83%
   workbooks_endpoint.py3542424 93%
TOTAL12000142788% 

@jorwoods

Copy link
Copy Markdown
Contributor

Should this use the config.CHUNK_SIZE_MB as the download size rather than a fixed 64mb?

Raises iter_content chunk size from 1024 to 65536 bytes in
_get_view_csv and _get_view_excel. The 1-KB default required
~200k iterations for a 200 MB download, causing apparent hangs
when callers materialise the full stream with b''.join(view.csv).

Closes #1374

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jacalata
jacalata force-pushed the jac/fix-large-view-download branch from 10703d8 to 695f784 Compare July 21, 2026 21:05
@bcantoni

Copy link
Copy Markdown
Contributor

Code Review

Summary: Raises the iter_content chunk size in ViewsEndpoint._get_view_csv / _get_view_excel from a hardcoded 1024 bytes to config.CHUNK_SIZE_MB * BYTES_PER_MB (50 MB default), fixing the effective hang reported in #1374 for large (~200 MB) view downloads.

Strengths

  • Correct root-cause fix — 1 KB chunks meant ~200k generator iterations for a 200 MB download; a much larger chunk size removes that overhead.
  • Reuses existing config infrastructure (config.CHUNK_SIZE_MB, BYTES_PER_MB) rather than a new magic number, matching the pattern already used in fileuploads_endpoint.py, datasources_endpoint.py, and custom_views_endpoint.py. This also gives users the existing TSC_CHUNK_SIZE_MB env var as a tuning knob.
  • Small, surgical change with no other behavior differences — low risk.
  • black --check passes on the changed file.

Issues / Gaps

Inconsistent chunk size left in place elsewhere. The same hardcoded 1024-byte pattern still exists, unfixed, in:

  • custom_views_endpoint.py:234 (_get_custom_view_csv) — directly parallel to _get_view_csv, the exact method being fixed here, and likely has the same large-download-hang risk.
  • flows_endpoint.py:235,242
  • datasources_endpoint.py:1069,1076
  • workbooks_endpoint.py:1131,1138

Worth confirming whether _get_custom_view_csv is intentionally out of scope for this PR, since it's the closest analog to the code being changed.

No test asserts the new chunk size. test_populate_csv, test_populate_csv_default_maxage, test_populate_excel, and test_filter_excel all mock small fixture files and only assert the reassembled content is correct — they'd pass identically whether the chunk size were 1024 bytes or 50 MB, so they don't protect this change from regressing. test_fileuploads.py:85-90 has a good existing pattern for this (sets TSC_CHUNK_SIZE_MB via env and asserts the actual chunk length) that could be adapted here, or alternatively mock/patch iter_content and assert it's called with the expected size.

The PR description's test plan checkbox is also still unchecked.

Correctness

config.CHUNK_SIZE_MB * BYTES_PER_MB = 50 * 1024 * 1024 ≈ 50 MB by default — consistent with the existing upload chunk-size default. No correctness or security concerns with the change itself.


🤖 Generated with Claude Code

@bcantoni

Copy link
Copy Markdown
Contributor

@jacalata kind of funny how claude code keeps finding issues with a claude code aided PR :) Consider doing those consistency points if they make sense.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issue with Downloading Large Views from Tableau Server to CSV

3 participants