Skip to content

[medium] Replace removed DataFrame.append in EVTX gap analyzer 🤖🤖🤖 - #3923

Open
elhoim wants to merge 2 commits into
google:masterfrom
elhoim:fix/t39-dataframe-append-removed-in-pandas-2-cra
Open

elhoim wants to merge 2 commits into
google:masterfrom
elhoim:fix/t39-dataframe-append-removed-in-pandas-2-cra

Conversation

@elhoim

@elhoim elhoim commented Oct 1, 2026 •

Copy link
Copy Markdown

BLUF

  • Priority: medium
  • Problem: The EVTX gap analyzer raises AttributeError whenever it finds gaps. win_evtxgap.py:258 calls DataFrame.append, which pandas 2.0 removed, and requirements.txt pins pandas>=3.0.3.
  • Fix: A one-line change from event_count.append(df_append, sort=False) to pd.concat([event_count, df_append], sort=False).
  • Test: TestEvtxGapPlugin.test_run_with_missing_days, a new end-to-end test of run().
  • Verified: The new test fails on master with the AttributeError and passes with the fix (module result: 2 passed, 2 skipped). black --check and pylint --rcfile=.pylintrc (Python 3.14, as in the CI matrix) are clean on both changed files.
  • What existing problem does this PR solve? The EVTX gap analyzer raises AttributeError whenever it finds gaps. win_evtxgap.py:258 calls DataFrame.append, which pandas 2.0 removed, and requirements.txt pins pandas>=3.0.3.
  • What new feature is being introduced with this PR? None, this is a bug fix.
  • Overview of changes to existing functions if required. A one-line change from event_count.append(df_append, sort=False) to pd.concat([event_count, df_append], sort=False).

Details

What was wrong

EvtxGapPlugin.run() builds a story when it finds both missing days and record-number gaps. It then pads the per-day counts with the missing days using DataFrame.append. On pandas 3.x that line raises:

AttributeError: 'DataFrame' object has no attribute 'append'

By that point the story text and the "Top EVTX Sources" aggregation are already saved. The run leaves a half-built story and a failed analysis. The "Event Records Per Day" chart and the record-number gap section, which is the evidence of log clearing, are never written.

This line runs only when both missing_ranges and record_gaps are non-empty, because of the early return before it. That condition holds for every run that produces a gap story, so all of those runs crashed.

Reproducing input

Security records #1 and #2 on 2020-01-01, and record #5 on 2020-01-03. The analyzer raises AttributeError instead of finishing the story.

Fix

pd.concat behaves the same way here. The sort_values("day") that follows restores the row order, and the extra timestamp column is still removed afterwards.

Regression test

The test mocks event_pandas and the sketch. The sketch mock needs id=1, because the story text formats self.sketch.id with {:d}. The test checks that run() reports gaps and that the manual_feed aggregation contains {"20200101": 2, "20200102": 0, "20200103": 1}.

Checks

  • All tests succeed.
  • Unit tests added.
  • e2e tests added.
  • Documentation updated.

Closing issues

No issue is linked to this PR yet.

@google-cla

google-cla Bot commented Oct 1, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@elhoim
elhoim force-pushed the fix/t39-dataframe-append-removed-in-pandas-2-cra branch from 10b1102 to 870d212 Compare October 1, 2026 07:08
EvtxGapPlugin.run() called event_count.append(), which pandas 2.0
removed; with pandas>=3.0.3 every run that found missing days raised
AttributeError after the story was half built. Use pd.concat instead
and add a run() regression test covering the missing-day path.
@elhoim
elhoim force-pushed the fix/t39-dataframe-append-removed-in-pandas-2-cra branch from 870d212 to 37cebdf Compare October 9, 2026 20:04

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant