fix(ogc): read each side of a "start/end" interval string like the list form (#443)

* fix(ogc): read each side of a "start/end" interval string like the list form

_format_api_dates sent any single string containing "/" unchanged. The
same range spelled as the documented string ("2024-01-01T10:00:00/..")
and as the list (["2024-01-01T10:00:00", None]) could select different
data: the string skipped the local-to-UTC conversion of naive times, the
offset-to-Z conversion, and the date truncation for date-only
collections, and "not-a-date/also-bad" reached the service unvalidated.

Split the string on its "/" and format each side as a list element is
formatted, so an unreadable side raises ValueError naming the caller's
argument before any request. An empty side becomes "..", and "../.."
means no filter, as [None, None] does. One side may still be an ISO 8601
duration paired with an instant ("2024-01-01/P7D"), kept unchanged; any
other shape (three sides, two durations, a duration with an open end)
raises ValueError naming the argument. A lone duration ("P7D") is
unchanged.

Follow-up named in #441.

* fix(ogc): reject duration sides, open starts per API, naive dates as UTC

- Reject a duration as either side of an interval string. The Water Data
  and NGWMN services reject '2024-01-01/P7D' and 'P7D/2024-01-08' with
  HTTP 400, and that error does not name the argument. _split_interval
  raises instead and passes the two sides to the list path as a tuple,
  which removes the duration handling in _format_interval and the
  _is_passthrough helper.
- Write an open range start in the form each API accepts. The NGWMN and
  STAC firewalls reject any datetime containing '../' with HTTP 403, so
  get_water_level(datetime=[None, end]) and get_ratings(time=[None, end])
  always failed. The Water Data API accepts '../end' and rejects '/end'
  with HTTP 400. OgcDialect.open_start holds the form (default '..');
  NGWMN sets '', and get_ratings passes '' to _format_api_dates because
  STAC has no dialect.
- Keep fractional seconds instead of truncating them, which changed a
  closed bound. The Water Data, NGWMN, and STAC services all accept them
  (checked 2026-10-12). Accept a minute-precision time and spaces around
  the '/'; both were previously sent unchanged and worked.
- Read a naive date or time as UTC, as pandas does and as
  get_nearest_continuous already did, instead of as the machine's local
  time. Convert an aware value to UTC, including in a date-only
  collection. Now that the string form is parsed like the list form,
  local time would have made time="2024-03-01/2024-03-02", the example
  in the docs, start at 06:00Z in Chicago and on the previous day in
  Sydney.
- Docstrings: get_ratings no longer says time is passed through
  unchanged, and the _format_api_dates notes describe open_start. The
  duration test ids name the side that is a duration.
- Date the NEWS entry by the day it lands and place it above #446's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HBt3Ee7eneKtdKQWq2T3YK

---------

Co-authored-by: thodson-usgs <thodson@usgs.gov>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Author: 3 people

NEWS.md

@@ -1,3 +1,5 @@
+**10/12/2026:** **Behavior change:** a date range passed as one `"start/end"` string, such as `time="2024-01-01T10:00:00/.."`, is now formatted like the list form `["2024-01-01T10:00:00", None]` in the `waterdata` and `ngwmn` OGC getters and in `waterdata.get_ratings()`. **Bug fix:** the string used to be sent unchanged, so the two forms could select different data: an offset was not converted to `Z`, and a date-only collection such as `daily` received times. A string now raises `ValueError` before any request is sent if a side is not a date (`time="2024-01-01/tomorrow"`), if it contains more than one `/`, or if either side is a duration (`"2024-01-01/P7D"`), which the services rejected with HTTP 400. An empty side is an open bound, and `"../.."` means no date filter. A lone duration such as `"P7D"` is unchanged. **Bug fix:** in `ngwmn.get_water_level()` and `waterdata.get_ratings()`, a range with an open start, such as `[None, "2016-01-01"]` or `"../2016-01-01"`, failed with HTTP 403 because those services' firewalls reject `../`. Both getters now send an empty start (`"/2016-01-01"`), which both services treat as open. The other Water Data getters still send `..`, because that API rejects an empty start. Fractional seconds, such as `2024-01-01T10:00:00.999Z`, are kept instead of truncated, and a minute-precision time (`2024-01-01T10:00Z`) and spaces around the `/` are accepted. **Behavior change:** a naive date or time is now UTC, as in pandas (`pd.to_datetime(..., utc=True)`) and `get_nearest_continuous()`, instead of the machine's local time, so `time=["2024-03-01", "2024-03-02"]` sends `2024-03-01T00:00:00Z/2024-03-02T00:00:00Z` on every machine. An aware value is converted to UTC, including in a date-only collection such as `daily`. For a local time, include the offset: `"2024-03-01T00:00:00-06:00"`.
+
 **10/12/2026:** Date arguments of the `waterdata` and `ngwmn` OGC getters (`time`, `begin`, `end`, `datetime`, `last_modified`) and of `waterdata.get_ratings()` accept `datetime.date`, `datetime.datetime`, and `pandas.Timestamp` values, alone or as either end of a range, such as `time=[df.index.min(), None]`. Each is read like the equivalent string: a `date` is midnight, and `NaT` is an open bound. **Bug fix:** these values used to raise `AttributeError` or `TypeError` without naming the argument. Any other type, such as a number, now raises `ValueError` naming the argument and the accepted types.
 
 **10/11/2026:** **Bug fix:** a list passed to most `waterdata` OGC filters, such as `get_daily(approval_status=["Approved", "Provisional"])`, returned no rows: the Water Data API reads a comma-joined value as one value for all but a few identifier and code filters, and rejects a numeric list with HTTP 400. Lists are now sent as a CQL2 `in` filter. A list combined with `filter=` in `get_monitoring_locations()` and `get_combined_metadata()` is no longer ignored. **Bug fix:** `filter_lang="cql-json"` failed with HTTP 400 in every `waterdata` OGC getter. The filter is now sent as the request body; one that is not a JSON object raises `ValueError`. A blank `filter=""` is now ignored rather than failing with HTTP 400. `ngwmn` is unchanged.

dataretrieval/ngwmn.py

@@ -80,12 +80,13 @@
 # (there is no service-specific id name as there is for the main collections).
 _NGWMN_OUTPUT_ID = "id"
 
-# NGWMN's request shape matches the generic OGC default: it reads comma-joined
-# GET values as lists, and its firewall refuses a text filter naming a site, so
-# a list could not be folded into one. It has no date-only collections, but its
-# result columns need their own coercion and sort vocabulary:
-# water-level observations are timestamped by ``sample_time`` (not the Water
-# Data ``time``) and report depths/levels in feet.
+# NGWMN reads comma-joined GET values as lists, and its firewall refuses a text
+# filter naming a site, so a list could not be folded into one. It has no
+# date-only collections, but its result columns need their own coercion and
+# sort vocabulary: water-level observations are timestamped by ``sample_time``
+# (not the Water Data ``time``) and report depths/levels in feet. Its firewall
+# rejects ``datetime=../2016-01-01`` with HTTP 403, so an open start is sent
+# empty (``/2016-01-01``), which NGWMN treats as open.
 NGWMN_DIALECT = OgcDialect(
     time_cols=frozenset({"sample_time"}),
     numerical_cols=frozenset(
@@ -96,6 +97,7 @@
         }
     ),
     sort_cols=("sample_time", "monitoring_location_id"),
+    open_start="",
 )
 
 

dataretrieval/ogc/dates.py

@@ -20,6 +20,8 @@
     "%Y-%m-%dT%H:%M:%S%z",
     "%Y-%m-%dT%H:%M:%S.%f",
     "%Y-%m-%dT%H:%M:%S",
+    "%Y-%m-%dT%H:%M%z",
+    "%Y-%m-%dT%H:%M",
     "%Y-%m-%d %H:%M:%S.%f",
     "%Y-%m-%d %H:%M:%S",
     "%Y-%m-%d",
@@ -118,14 +120,14 @@ def _format_one(dt: _DateLike, *, date: bool, name: str) -> str:
     if dt is None or _is_blank(dt):
         return _OPEN_BOUND
     parsed = _to_datetime(dt, name=name)
+    # As in pandas, a naive value is UTC and an aware one is converted to UTC,
+    # so the same call sends the same range on every machine.
+    utc = parsed if parsed.tzinfo is None else parsed.astimezone(ZoneInfo("UTC"))
     if date:
-        return parsed.strftime("%Y-%m-%d")
-    # Naive inputs are interpreted in the system local zone (for backwards
-    # compatibility). Use ``.astimezone()`` rather than a fixed offset so each
-    # value is resolved against the DST rules for its own date — a fixed
-    # ``datetime.now()`` offset shifted off-season inputs by an hour.
-    aware = parsed if parsed.tzinfo is not None else parsed.astimezone()
-    return aware.astimezone(ZoneInfo("UTC")).strftime("%Y-%m-%dT%H:%M:%SZ")
+        return utc.strftime("%Y-%m-%d")
+    # Keep fractional seconds: truncating them changes a closed bound.
+    fraction = f".{utc.microsecond:06d}".rstrip("0") if utc.microsecond else ""
+    return f"{utc.strftime('%Y-%m-%dT%H:%M:%S')}{fraction}Z"
 
 
 def _coerce_to_list(
@@ -147,9 +149,27 @@ def _coerce_to_list(
     return list(datetime_input)
 
 
-def _is_passthrough(single: str) -> bool:
-    """True when a single-element input should be returned as-is."""
-    return bool(_DURATION_RE.match(single) or "/" in single)
+def _split_interval(interval: str, *, name: str) -> tuple[str, str]:
+    """Split a ``"start/end"`` string into its two sides.
+
+    Each side is then formatted like a list element, so
+    ``"2024-01-01T10:00:00/.."`` sends the same range as
+    ``["2024-01-01T10:00:00", None]``.
+
+    Raises ``ValueError`` naming *name* unless the string is two sides
+    separated by one ``"/"``, neither of them a duration. The Water Data and
+    NGWMN services reject ``"2024-01-01/P7D"`` and ``"P7D/2024-01-08"`` with
+    HTTP 400.
+    """
+    sides = [side.strip() for side in interval.split("/")]
+    if len(sides) != 2 or any(_DURATION_RE.match(side) for side in sides):
+        raise ValueError(
+            f"{name} is not a valid interval: {interval!r}. Pass a start and "
+            "an end separated by '/', such as '2024-01-01/2024-12-31', with "
+            "'..' for an open end ('2024-01-01/..'). Neither side can be a "
+            "duration."
+        )
+    return sides[0], sides[1]
 
 
 def _all_blank(items: list[_DateLike]) -> bool:
@@ -163,6 +183,7 @@ def _format_api_dates(
     *,
     name: str = "date input",
     single_value_hint: str = "an instant or a duration ('2020-01-01', 'P7D')",
+    open_start: str = _OPEN_BOUND,
 ) -> str | None:
     """
     Formats date or datetime input(s) for use with an API.
@@ -194,6 +215,12 @@ def _format_api_dates(
         forms (``get_ratings`` rejects durations) enforces that itself and
         passes a hint naming only what it accepts, so the remedy does not
         direct a caller to a value that is rejected.
+    open_start : str, optional
+        The string written for an open range start (``".."`` by default).
+        OGC API Features and STAC also allow an empty string
+        (``"/2024-01-01"``), which an API whose firewall rejects ``"../"``
+        requires; see :attr:`OgcDialect.open_start`. An open end is always
+        ``".."``.
 
     Returns
     -------
@@ -213,20 +240,25 @@ def _format_api_dates(
     Notes
     -----
     - A single blank/NA value returns None. In a two-value range, a blank/NA
-      or ``".."`` endpoint is rendered as ``".."`` to denote an open bound
-      (e.g. ``"2024-01-01/.."``); the range is only None when *every* element
-      is blank/NA/``".."``.
-    - Supports ISO 8601 durations such as "P7D" and "PT36H" and pre-formatted
-      intervals containing ``"/"``; both are passed through unchanged.
+      or ``".."`` endpoint is an open bound. An open end is written ``".."``
+      (``"2024-01-01/.."``) and an open start as ``open_start``. The range is
+      None only when *every* element is blank/NA/``".."``.
+    - Supports ISO 8601 durations such as "P7D" and "PT36H", which are passed
+      through unchanged.
+    - A single string containing ``"/"`` is an interval: it is split into the
+      two-value form, so ``"2024-01-01/.."`` and ``["2024-01-01", None]`` are
+      formatted alike. A duration is not accepted as either side.
     - Converts datetimes to UTC and formats as ISO 8601 with 'Z' suffix when
-      `date` is False. Inputs with an explicit offset (``Z`` or ``+HH:MM``) are
-      converted from that offset to UTC; naive inputs are interpreted in the
-      local time zone for backwards compatibility.
+      `date` is False. As in pandas, an input with an offset (``Z`` or
+      ``+HH:MM``) is converted to UTC and a naive one is UTC; a date-only
+      collection gets the UTC date.
     """
     if datetime_input is None:
         return None
 
     items = _coerce_to_list(datetime_input, name)
+    if len(items) == 1 and isinstance(items[0], str) and "/" in items[0]:
+        items = list(_split_interval(items[0], name=name))
 
     if _all_blank(items):
         return None
@@ -238,8 +270,11 @@ def _format_api_dates(
             "or two for a closed interval ('2020-01-01', '2020-12-31')."
         )
 
-    # Pass through duration ("P7D", "PT36H") and pre-formatted interval ("a/b")
-    if len(items) == 1 and isinstance(items[0], str) and _is_passthrough(items[0]):
+    # Pass through a duration ("P7D", "PT36H")
+    if len(items) == 1 and isinstance(items[0], str) and _DURATION_RE.match(items[0]):
         return items[0]
 
-    return "/".join(_format_one(dt, date=date, name=name) for dt in items)
+    formatted = [_format_one(dt, date=date, name=name) for dt in items]
+    if len(formatted) == 2 and formatted[0] == _OPEN_BOUND:
+        formatted[0] = open_start
+    return "/".join(formatted)

dataretrieval/ogc/policy.py

@@ -61,13 +61,20 @@ class OgcDialect:
         Columns to sort the combined result by, in priority order. Sorting
         is applied only when the first (primary) column is present; any
         later columns also present are added as secondary keys.
+    open_start : str
+        The string written for the open start of a date range: ``".."``
+        (``"../2024-01-01"``, the default) or ``""`` (``"/2024-01-01"``).
+        OGC API Features allows both, but each service accepts only one: the
+        Water Data API rejects an empty start with HTTP 400, and the NGWMN
+        firewall rejects ``"../"`` with HTTP 403.
     """
 
     lists_as_cql2: bool = False
     date_only_services: frozenset[str] = field(default_factory=frozenset)
     time_cols: frozenset[str] = field(default_factory=frozenset)
     numerical_cols: frozenset[str] = field(default_factory=frozenset)
     sort_cols: tuple[str, ...] = field(default_factory=tuple)
+    open_start: str = ".."
 
 
 # Default dialect: a plain OGC API, with lists comma-joined in the URL and every

dataretrieval/ogc/requests.py

@@ -211,6 +211,7 @@ def _construct_api_requests(
                 date=(
                     collection in dialect.date_only_services and key != "last_modified"
                 ),
+                open_start=dialect.open_start,
             )
     json_filter = _pop_json_filter(kwargs)
     params, post_params = _partition_request_params(

dataretrieval/waterdata/ratings.py

@@ -82,10 +82,11 @@ def get_ratings(
         persisting the bytes to disk; ``df.attrs["url"]`` still records
         where each rating came from.
     time : string, date, datetime, or pandas.Timestamp, or a list of them, optional
-        STAC ``datetime`` filter (passed through verbatim under that name)
-        — a single date / datetime, an interval (``"start/end"``,
-        optionally half-bounded with ``..``), or a list of two values
-        with ``None`` for an open end (``["2026-04-29", None]``). A value
+        STAC ``datetime`` filter — a single date / datetime, an interval
+        (``"start/end"``, optionally half-bounded with ``..``), or a list of
+        two values with ``None`` for an open end (``["2026-04-29", None]``).
+        An interval string is read like the list form: each side is
+        converted to UTC, and a naive one is UTC, as in pandas. A value
         that is not a date raises ``ValueError``. ISO 8601 *durations*
         (``"P1M"``, ``"PT36H"``, …) are **not** supported by the
         rating-curve service; passing one raises ``ValueError``.
@@ -179,7 +180,12 @@ def get_ratings(
     _validate_time_no_duration(time)
     time_str = (
         _format_api_dates(
-            time, name="time", single_value_hint="an instant ('2020-01-01')"
+            time,
+            name="time",
+            single_value_hint="an instant ('2020-01-01')",
+            # The STAC firewall rejects ``datetime=../2026-01-01`` with HTTP
+            # 403; STAC treats an empty start (``/2026-01-01``) as open.
+            open_start="",
         )
         if time is not None
         else None

docs/source/userguide/timeconventions.rst

@@ -10,6 +10,12 @@ continuous (instantaneous) values — ``time`` is a timezone-aware
 ``datetime64[us, UTC]`` column. Daily values represent a whole calendar day,
 so their ``time`` column is timezone-naive (dates only).
 
+The same rule applies to a getter's date arguments (``time``, ``begin``,
+``end``, ``datetime``, ``last_modified``). As in pandas, a date or time without
+an offset is UTC, and one with an offset is converted to UTC, so
+``time="2024-03-01/2024-03-02"`` selects the same UTC days on every machine.
+For a local time, include the offset: ``"2024-03-01T00:00:00-06:00"``.
+
 
 Inspecting Timestamps
 *********************

tests/ngwmn_test.py

@@ -447,8 +447,6 @@ def test_get_water_level_datetime_subsets(httpx_mock):
     windowed_body = _collection(_WATER_LEVELS["features"][1:])
     _mock(httpx_mock, "waterLevelObs", windowed_body)
 
-    # Offsets given explicitly: a naive input would be resolved against the
-    # machine's local zone, making the expected wire value machine-dependent.
     windowed, _ = ngwmn.get_water_level(
         monitoring_location_id=_SITE,
         datetime=["2022-01-01T00:00:00Z", "2024-01-01T00:00:00Z"],
@@ -460,6 +458,20 @@ def test_get_water_level_datetime_subsets(httpx_mock):
     assert qs["datetime"] == ["2022-01-01T00:00:00Z/2024-01-01T00:00:00Z"]
 
 
+@pytest.mark.parametrize(
+    "datetime", [[None, "2016-01-01T00:00:00Z"], "../2016-01-01T00:00:00Z"]
+)
+def test_get_water_level_sends_an_open_start_empty(httpx_mock, datetime):
+    """The NGWMN firewall rejects ``datetime=../2016-01-01`` with HTTP 403, so
+    an open start is sent as an empty string, which NGWMN treats as open."""
+    _mock(httpx_mock, "waterLevelObs", _WATER_LEVELS)
+
+    ngwmn.get_water_level(monitoring_location_id=_SITE, datetime=datetime)
+
+    qs = _queries(httpx_mock, "waterLevelObs")[0]
+    assert qs["datetime"] == ["/2016-01-01T00:00:00Z"]
+
+
 def test_get_lithology(httpx_mock):
     _mock(httpx_mock, "lithologyObs", _LITHOLOGY)
 

tests/waterdata_ratings_test.py

@@ -171,6 +171,23 @@ def test_get_ratings_download_and_parse_false_returns_features(httpx_mock):
     assert features[0]["id"] == "USGS-01104475.exsa.rdb"
 
 
+@pytest.mark.parametrize(
+    "time", [[None, "2026-04-29T00:00:00Z"], "../2026-04-29T00:00:00Z"]
+)
+def test_get_ratings_sends_an_open_start_empty(httpx_mock, time):
+    """The STAC firewall rejects ``datetime=../2026-04-29`` with HTTP 403, so an
+    open start is sent as an empty string, which STAC treats as open."""
+    httpx_mock.add_response(
+        method="GET", url=STAC_SEARCH_RE, json=_stub_search_response()
+    )
+    get_ratings(
+        monitoring_location_id="USGS-01104475", time=time, download_and_parse=False
+    )
+    (request,) = httpx_mock.get_requests()
+    params = parse_qs(urlsplit(str(request.url)).query)
+    assert params["datetime"] == ["/2026-04-29T00:00:00Z"]
+
+
 def test_get_ratings_keeps_a_dotdot_open_bound_in_time(httpx_mock):
     """``time=[start, ".."]`` is the documented open-ended range. The ``..``
     endpoint used to fail to parse, which discarded the whole range, so the
@@ -182,8 +199,6 @@ def test_get_ratings_keeps_a_dotdot_open_bound_in_time(httpx_mock):
     )
     get_ratings(
         monitoring_location_id="USGS-01104475",
-        # A UTC instant, so the expected bound does not depend on the local zone
-        # a naive date is read in.
         time=["2026-04-29T00:00:00Z", ".."],
         download_and_parse=False,
     )

tests/waterdata_test.py

@@ -721,6 +721,15 @@ def test_get_args_materializes_numpy_and_series_numeric_params():
     assert "bbox=-92.8%2C44.2%2C-88.9%2C46.0" in str(req.url)
 
 
+def test_construct_api_requests_keeps_dotdot_for_an_open_start():
+    """The Water Data API rejects an empty start (``time=/2024-01-31``) with
+    HTTP 400, so its dialect keeps the default ``..``."""
+    req = _construct_api_requests(
+        "daily", monitoring_location_id="USGS-05427718", time=[None, "2024-01-31"]
+    )
+    assert "time=..%2F2024-01-31" in str(req.url)
+
+
 def test_construct_api_requests_two_element_date_list_becomes_interval():
     """A two-element date list is interpreted as start/end of an OGC datetime
     interval (joined with '/'), NOT as two discrete dates. The OGC `datetime`
@@ -948,7 +957,9 @@ def test_get_daily_keeps_a_dotdot_open_bound_in_time(httpx_mock):
 
 
 @pytest.mark.parametrize(
-    "time", ["2025-13-45", ["2025-01-01", "yesterday"]], ids=["single", "range"]
+    "time",
+    ["2025-13-45", ["2025-01-01", "yesterday"], "not-a-date/also-bad"],
+    ids=["single", "range", "interval_string"],
 )
 def test_get_daily_rejects_an_unreadable_time_before_any_request(httpx_mock, time):
     """A bound that matches no date format used to drop the whole filter, so