[maresb]

I am trying to combine some massive S3-based Zarr references. A MultiZarrToZarr operation which would otherwise take a minute would take days because the size of each file on S3 is being queried. Thus I want to be able to pass in inline_threshold=None so that my datasets combine in a reasonable timeframe.

[martindurant]

I think this was already possible with inline_threashold=0 ?

[maresb]

Nope, if inline_threshold=0 then fs.info(fn) still gets called in order to ensure that the file size isn't negative. 😅

We could change this so that we use 0 in place of None. That would probably make more sense actually. Then my conditional self.inline is not None becomes self.inline > 0.

[martindurant]

That sounds like a good plan

[maresb]

Ok @martindurant, that should take care of it.

Now the code should be functionally identical to the previous, except that the file size lookup is skipped when unneeded. In other words, no more inline_threshold=None but instead inline_threshold=0 is optimized.

[maresb]

Oh, I'm sorry, I'm on my laptop at the moment and it looks like I forgot to set up pre-commit 🙈

Please feel free to squash this PR.

[maresb]

Hmm, this test failure I can reproduce locally, also from main. I'm guessing it's a dependency issue.

[martindurant]

@Anu-Ra-g , any idea?

>       assert (
            idx_df["typeOfLevel"][[42, 46, 49, 50]]
            == ["heightAboveGround", "isobaricInhPa", "surface", "heightAboveGround"]
        ).all()
E       AssertionError: assert False
E        +  where False = <bound method NDFrame._add_numeric_operations.<locals>.all of 42     True\n46     True\n49     True\n50    False\nName: typeOfLevel, dtype: bool>()
E        +    where <bound method NDFrame._add_numeric_operations.<locals>.all of 42     True\n46     True\n49     True\n50    False\nName: typeOfLevel, dtype: bool> = 42    heightA... dtype: object == ['heightAbove...tAboveGround']
E             Full diff:
E             - [
E             -  'heightAboveGround',
E             ? ^^                 --
E             + 42    heightAboveGround
E             ? ^^^^^^
E             -  'isobaricInhPa',...
E
E             ...Full output truncated (8 lines hidden), use '-vv' to show.all

[maresb]

Interesting, it seems like something's getting mangled. When I run locally the assert above passes for me, but the subsequent one fails.

Note that for me stepType is instant instead of the asserted accum.

idx_df.iloc[83]
varname                                                             wz
typeOfLevel                                              isobaricInhPa
stepType                                                       instant
name                                       Geometric vertical velocity
isobaricInhPa                                                    975.0
step                                                   0 days 06:00:00
time                                               2023-09-28 00:00:00
valid_time                                         2023-09-28 06:00:00
uri                            ~/repos/kerchunk/kerchunk/tests/gfs....
offset                                                        21234675
length                                                         1035696
inline_value                                                      None
surface                                                            NaN
heightAboveGround                                                  NaN
meanSea                                                            NaN
Name: 83, dtype: object

[maresb]

I'm guessing it's a dependency issue.

The culprit is eccodes v2.38.0.

[a-nu-ragn]

@maresb I tried to emulate the above steps on my local system. The error is not occurring consistently.
And this object is still accum when I tried it locally,

varname                                                           watr
typeOfLevel                                                    surface
stepType                                                         accum
name                                                      Water runoff
isobaricInhPa                                                      NaN
step                                                   0 days 06:00:00
time                                               2023-09-28 00:00:00
valid_time                                         2023-09-28 06:00:00
uri                  ./kerchunk/tests/gfs.t00z.pgrb2.0p25.f006.test...
offset                                                        59215098
length                                                          348767
inline_value                                                      None
surface                                                            0.0
heightAboveGround                                                  NaN
meanSea                                                            NaN
Name: 83, dtype: object

Even though the error is occuring, the remaining steps work fine.

[maresb]

Thanks @Anu-Ra-g! May I ask which version of eccodes you are using?

[a-nu-ragn]

I'm on eccodes version 2.36.0

[maresb]

Since the test failure is unrelated to this PR, I recommend moving discussion to #508.

@Anu-Ra-g, to reproduce the failure you need to upgrade to eccodes v2.38.0.

[maresb]

Thanks a lot @martindurant for the merge!!! 🙏