PDEP-14: Dedicated string data type for pandas 3.0

[jorisvandenbossche]

Following the discussion in #57073, this proposes a possible solution to get a string dtype in pandas 3.0 (essentially writing out my compromise attempt at #57073 (comment) as a formal proposal).
This also covers the issue tracking the required work for the string dtype in #54792.

Abstract

This PDEP proposes to introduce a dedicated string dtype that will be used by default in pandas 3.0:

• In pandas 3.0, enable a "string" dtype by default, using PyArrow if available or otherwise the numpy object-dtype alternative.
• The default string dtype will use missing value semantics using NaN consistent with the other default data types.

This will give users a long-awaited proper string dtype for 3.0, while 1) not (yet) making PyArrow a hard dependency, but still a dependency used by default, and 2) leaving room for future improvements (different missing value semantics, using NumPy 2.0 or nanoarrow, etc).

Sub-discussions:

• Default string dtype (PDEP-14): naming convention to distinguish the dtype variants #58613

cc @pandas-dev/pandas-core @pandas-dev/pandas-triage

[bashtage]

A good attempt at providing the compromise that is being asked for.

Some possible names that spring to mind: pyarrow_legacy, pyarrow_nan

[bashtage]

Should you allow the possability of a NumPy 2 improved type for pandas 3? With a heirarchy arrow -> np 2 -> np object?

[jorisvandenbossche]

This proposal does not preclude any further improvements for the numpy-based string dtype using numpy 2.0. A few lines below I explicitly mention it as a future improvement and in the "Object-dtype "fallback" implementation" section as well.

I just don't want to explicitly commit to anything for pandas 3.0 related to that, given it is hard to judge right now how well it will work / how much work it is to get it ready (not only our own implementation, but also support in the rest of the ecosystem). If it is ready by 3.0, then we can evaluate that separately, but this proposal doesn't stand or fall with it.

Regardless of whether to also use numpy 2.0, we have to agree on 1) making a "string" dtype the default for 3.0, 2) the missing value behaviour to use for this dtype, and 3) whether to provide an alternative for PyArrow (in which case we need the object-dtype version anyway since we also can't require numpy 2.0). I would like the proposal to focus on those aspects.

[bashtage]

Is it worth mentioning why this has been objected to? As far as I am aware virtually all objections are due to the installation size effect, and not performance or compatibility.

[jorisvandenbossche]

I can certainly mention something, but would prefer to keep that brief to focus here on the strings context and not trigger discussion here about the merits of those objections.
(for example, it's not only installation size, but also the difficulty to install from source in case there are no wheels)

[jorisvandenbossche]

Added "(mostly around installation complexity and size)"

[WillAyd]

I don't think NumPy 2.0 will reduce the need to make pyarrow a dependency for strings; as far as I am aware it is not natively returned by any I/O operation and it has a completely different string architecture than pyarrow, so there is no zero-copy capability. Those seem like they either will require a large amount of string copying or a hefty amount of updates to make it natively work with our I/O, as well as with the larger Arrow ecosystem. That's a huge amount of things to gloss over

[jorisvandenbossche]

I don't think NumPy 2.0 will reduce the need to make pyarrow a dependency for strings

I think it can do that if your motivation for wanting pyarrow is the better performance compared to object-dtype. In that case, numpy 2.0's StringDType can give you a part of the speedup, without requiring pyarrow.
The discussion in #57073 also started from that point of view, mentioning numpy 2.0 as an alternative to requiring pyarrow, so based on that my feeling is that what I wrote here is correct (or at least seen as such by some people).

But you are completely right that there are a lot of things that would need to be implemented to make it fully usable for us. That's also the reason that this PDEP does not say to use numpy 2.0, but defers that as a possible future enhancement, to discuss later. And you are also right that it has drawbacks compared to a Arrow based solution (using Arrow memory layout, but not necessary using pyarrow the package), another reason for me personally to again defer that to a separate discussion.

I just wanted to mention it for the complete context of the string dtype history and discussion. Now, I already mention its existence in the previous paragraph, so could keep it shorter here.
(and if you have any concrete suggestions to word this better, I am all ears!)

[WillAyd]

So we are reusing pd.StringDtype() in this case right? Is that going to break existing use cases where users have relied on that using pd.NA as a sentinel?

[jorisvandenbossche]

So we are reusing pd.StringDtype() in this case right?

Yes, and that is what already happens since pandas 2.1 with future.infer_string enabled

Is that going to break existing use cases where users have relied on that using pd.NA as a sentinel?

Yes, I mentioned that in the "Backwards compatibility" section

[WillAyd]

Ah thanks - sorry for overlooking that. So I think it goes without saying then that if we go this route we no longer will declare pd.StringDtype() experimental? Or are we still trying to keep that reservation knowing even this is not considered a long term design decision?

[jorisvandenbossche]

So I think it goes without saying then that if we go this route we no longer will declare pd.StringDtype() experimental?

Yep, given the proposal is to enable this by default, I think that is indeed saying to remove the experimental label (I can mention that somewhere explicitly if that helps)

Or are we still trying to keep that reservation knowing even this is not considered a long term design decision?

Once we have a "string", we will always have one, I think. That aspect is the long term decision this PDEP is proposing. We might change later the missing value semantics, but that doesn't mean the string dtype proposed here is still experimental (just like our default "int64" dtype is not experimental). At the time that we would decide to enable new missing value semantics by default, then "string" will "simply" start meaning something differently.

[jbrockmendel]

ValueError: Could not find PDEP number in 'PDEP: Dedicated string data type for pandas 3.0'. Please make sure to write the title as: 'PDEP-num: PDEP: Dedicated string data type for pandas 3.0'.

[jbrockmendel]

StringDtype(storage="pyarrow", semantics="numpy")? or instead of semantics, could use "na_value=np.nan`

[simonjayhawkins]

If i'm understanding correctly about the motivation for the change in dtype (improved overall user experience), then moving forward I suspect that when we can have improved/native dtypes for other data types (nested, date, etc) that the same logic would need to apply, i.e. we would need to have a variants of these with NumPy semantics.

Now this probably falls under PDEP-13 but if we have semantics as a argument (that users would see and use) we could still end up with columns using different missing value indicators?

[simonjayhawkins]

StringDtype(storage="pyarrow", semantics="numpy")? or instead of semantics, could use "na_value=np.nan`

or maybe "nullable=[True|False]"

However, at the moment, we distinguish the nullable data types for the other dtypes (int, float, etc) with capitalization and so for consistency could also consider string/String as the dtypes.

[WillAyd]

PDEP-13 proposes StringDtype(backend="pyarrow", na_marker=np.nan). I think the repr should just be updated to reflect that; trying to sift through the meaning of int versus Int versus int[pyarrow] compared to string versus string[pyarrow] versus string[pyarrow_numpy] I think would be a distraction for this proposal

[jorisvandenbossche]

StringDtype(storage="pyarrow", semantics="numpy")? or instead of semantics, could use "na_value=np.nan`

@jbrockmendel good point that we can also use other keywords than just storage to make the distinction

if we have semantics as a argument (that users would see and use) we could still end up with columns using different missing value indicators?

Only if users explicitly specify a non-default value for this, and never by default. This is the same with whatever option we come up with (eg also when using dtype_backend="pyarrow" or explicitly asking for one of the masked dtypes with dtype=Int64 or .. you can end up with a DataFrame with columns with mixed semantics)

we distinguish the nullable data types for the other dtypes (int, float, etc) with capitalization and so for consistency could also consider string/String as the dtypes.

Yeah, only unfortunately to be consistent with the other dtypes where we use capitalization, it would need to be "string" for the new NaN-based dtype, and "String" for the "nullable" NA-based variant. And so that doesn't help with backwards compatibility, because "string" right now means the nullable dtype. Given that, I would personally not use capitalization here (which also only is a solution for the string alias naming, not for the StringDtype(..) API)

---

To keep the sub-discussions manageable, I moved this specific topic out of this inline comment thread, and into it's own issue: #58613

[simonjayhawkins]

I see no reason not to use #57073 as the discussion issue as any further discussion will be here and #57073 can now focus on whether to reject PDEP-10 and what to do about the planned improvements to other dtypes.

My assumption is that approval of this PDEP should not, in itself, be a justification to overturn the PDEP-10 decision even though they are very much related and the implementation of the fallback option is only applicable if PDEP-10 is formally rejected.

[rhshadrach]

@jorisvandenbossche - I've renamed this PDEP-14 to fix the doc build job. The docs build automatically picks up added PDEP PRs for the website, and they need a number for that to succeed.

[lithomas1]

The pyarrow_numpy StringArray also returns numpy arrays as results for some operations.

I think this is also important to mention.

[jorisvandenbossche]

At this point, I haven't yet mentioned that the original StringDtype returns masked arrays from operations (only that it uses pd.NA). I only mention that when going more in detail on this topic in the "Missing value semantics" subsection. Given that, I would also leave it here to the generic "missing value semantics" for the new variant as well (to not make the background section even longer. I can certainly expand the "Missing value semantics" section if needed)

[lithomas1]

It would be nice to clarify that this is a separate dtype from the original string[python] dtype, just to make it clear that the original StringDtype is not changing (and still will return masked arrays, and use pd.NA as its missing sentinel)

[jorisvandenbossche]

I tried to clarify in the test that it is indeed a new variant of the string dtype and uses a subclass to reuse most code

[lithomas1]

I would drop this bit about nanoarrow (given it is not explained/introduced in the paragraphs beforehand).

If you want to add an explanation above, that's also fine with me.

[jorisvandenbossche]

I added a link to the discussion issues for both numpy 2.0 and nanoarrow, so people can find more explanation there if they want.

[lithomas1]

This part of the plan worries me a little.

Maybe it would be better to cut off a 2.3 from 2.2.x.

I think there's a significant proportion of the downloads for 2.2 that aren't on the latest patch release.
I think there's ~ 1/3 of the downloads that are fetching 2.2.0.

[lithomas1]

Also,
it would be good to mention which version of pandas is expected to have infer_string be able to infer to the object fallback option.

[simonjayhawkins]

a 2.3 release (maybe around the same time as 3.0rc) sounds reasonable.

If the features/bugfixes added to 2.3 are limited to the string dtype then we shouldn't need many patch releases. We may not need to fix any string dtype related issues that are fixed for 3.0 as these will be behind a flag in 2.3 and so shouldn't break existing code.

On the other hand, as these features are behind a flag, maybe releasing a 2.3 would not gain the field testing we hope for.

And therefore, instead of doing a 2.3, planning for at least a couple of release candidates for 3.0 would better achieve this.

[lithomas1]

@jorisvandenbossche

Thoughts on this?

[jorisvandenbossche]

Maybe it would be better to cut off a 2.3 from 2.2.x.

Yes, if we still plan to add a deprecation warning and change the naming scheme in StringDtype, calling that 2.3.0 sounds as the best option (I had been planning to propose doing a 2.3.0 (from the 2.2.x branch) anyway to bump the warning for CoW from DeprecationWarning to FutureWarning)

[lithomas1]

I don't think we can just claim this. I don't disagree, but this should be backed up more.

At least from the feedback received from #57073 and the other issue, there's at least a significant part of the user base that doesn't use strings.

There's also a significant chunk of the population that can't install pyarrow (due to size requirements or exotic platforms or whatever).

[WillAyd]

I am not sure this argument is that convincing either, although for slightly different reasons. I don't think we need to feel rushed for the next release

[jorisvandenbossche]

I don't think we can just claim this. I don't disagree, but this should be backed up more.

@lithomas1 can you clarify which part of the paragraph you think requires more backing up?
The fact that I say a "significant" part of our user base has pyarrow installed?

I don't think we can ever know exact numbers for this, but one data point is that pandas currently has 210M monthly downloads and pyarrow has 120M monthly downloads. Of course not all of those pyarrow users are also using pandas, but let's just assume that half of those pyarrow downloads come from people using pandas, that would mean that around 30% for our users already have pyarrow installed, which I would consider as a "significant part".
(and my guess is that for people working with larger datasets, where the speed of pyarrow becomes more important, this percentage will be higher, for example because of using the parquet IO)

But anyway, we are never going to know this exact number, but IMO we do know that a significant part of our userbase has pyarrow and will benefit from using that by default.

there's at least a significant part of the user base that doesn't use strings.

Yes, and then this PDEP is not relevant for them. But it's not because some users don't use strings, that we shouldn't improve the life of those users that do use strings? (so just not really understanding how this is a relevant argument)

There's also a significant chunk of the population that can't install pyarrow

Yes, and this PDEP addresses that by allowing a fallback when pyarrow is not installed.

I am not sure this argument is that convincing either, although for slightly different reasons.

@WillAyd can you then clarify which other reasons?

[WillAyd]

My other reason is that I don't think there is ever a rush to get a release out; we have historically never operated that way

[jorisvandenbossche]

I don't think there is ever a rush to get a release out; we have historically never operated that way

For the last six years, we have roughly released a new feature release every six months. We indeed never rush a specific release if there is something holding it up for a bit, but historically we have been releasing somewhat regularly.

At this point, a next feature release will be 3.0 given the amount of changes we already made on the main branch that require the next release cut from main to be 3.0 and not 2.3 (enforced deprecations etc).
(we can cut a 2.3 release from the the 2.2.x maintenance branch, which we might want to do for several reasons, but not counting that as a feature release for this discussion, as that will not actually contain features)

So I would say there is not necessarily a rush to do a release with a default "string" dtype (that is up for debate, i.e. this PDEP), but there is some rush to get a 3.0 release out. In the meaning that I think we don't want to delay 3.0 for like half a year or longer.

So for me delaying the string dtype, essentially means not including it in 3.0 but postponing it to pandas 4.0 (I should maybe be clearer in the paragraph above about that).

And then I try to argue in the text here that postponing it for 4.0 has a cost (or, missed benefit), because we have an implementation we could use for a default string dtype in pandas 3.0, and postponing introducing it makes that users will use the sub-optimal object dtype for longer, for (IMO) no good reason.

[lithomas1]

I don't think we can just claim this. I don't disagree, but this should be backed up more.

@lithomas1 can you clarify which part of the paragraph you think requires more backing up? The fact that I say a "significant" part of our user base has pyarrow installed?

It'd be nice to add how much perf benefits Arrow strings are expected to bring (e.g. 20%? 2x? 10x?).
Putting in the part about how many users have pyarrow would also help.

It'd also be good to elaborate on the usability part. IIUC, the main benefit here is not having to manually check element to see whether your object dtype'd column contains strings (since I think all the string methods work on object dtype'd columns).

I think it's also fair to amend this part to say "massive benefits to users that use strings" (instead of in general).

[WillAyd]

Benchmarks are going to be highly dependent on usage and context. If working in an Arrow native ecosystem, the speedup of strings may be a factor over 100x. If working in a space where you have to copy back and forth a lot with NumPy, that number goes way down.

I think trying to set expectations on one number / benchmark for performance is futile, but generally Arrow only helps, and makes it so that we as developers don't need to write custom I/O solutions (eg: ADBC Drivers, parquet, read_csv with pyarrow all work with Arrow natively with no extra pandas dev effort)

[jorisvandenbossche]

It'd be nice to add how much perf benefits Arrow strings are expected to bring (e.g. 20%? 2x? 10x?).

Benchmarks are going to be highly dependent on usage and context.

Indeed, for single operations you can easily get a >10x speedup, but of course a typical workflow does not consist of just string operations, and the overall speedup depends a lot (see this slide for one small example comparison (https://phofl.github.io/pydata-berlin/pydata-berlin-2023/intro.html#74) and this blogpost from Patrick showing the benefit in a dask example workflow (https://towardsdatascience.com/utilizing-pyarrow-to-improve-pandas-and-dask-workflows-2891d3d96d2b).

but generally Arrow only helps, and makes it so that we as developers don't need to write custom I/O solutions

That is often true, but except for strings ;).
For strings, the faster compute kernels will still give a lot of value even if your IO wasn't done through Arrow (and give a lot more value compared to using pyarrow for numeric data)

[simonjayhawkins]

Thanks @jorisvandenbossche for the PDEP.

I am generally in agreement with the motivation for this PDEP on the proviso that any approval is not rejecting PDEP-10. The motivation of accepting PDEP-10 by the team members could have been related to the perceived maintenance burden, a more performant string dtype, interoperability, having better default inference for other data types or maybe some other reason. This current PDEP only addresses one aspect of that decision.

One other aspect that is not mentioned here and was not mentioned in PDEP-10 is the consequences of choosing PyArrow as a backend. Bearing in mind, that it was felt that the implications of using nullable semantics for default dtypes was not discussed, I wonder whether we should have a section that discusses the other implications of choosing PyArrow in this PDEP, e.g. implications of choosing 1d immutable arrays as the backend.

[simonjayhawkins]

and do we consider adding a performance warning to the fallback also?

[jorisvandenbossche]

and do we consider adding a performance warning to the fallback also?

I personally wouldn't do that always / for each method, because that would be super noisy (and in some cases, like smallish data, it doesn't matter that much, so getting those warnings would be annoying).

If we wanted to warn users to gently push them towards installing pyarrow, I think we could do a warning but only 1) raise it once, and 2) only when doing one of the string operations on a big enough dataset (with some threshold).

Now, your question reminds me that the current pyarrow-backed string dtype has those fallback warnings for very specific cases, which I personally think we should stop doing when it becomes the default dtype. Given this is already for the existing implementation (and to keep the many discussion lines here a bit more limited), I opened a separate issue for this: #58581.
(but if there is agreement on that other issue, can of course briefly mention that here later)

[simonjayhawkins]

fair point. from the recent user feedback of adding the deprecation warning for the PyArrow requirement, then maybe not having any warnings is wise.

that the current pyarrow-backed string dtype has those fallback warnings for very specific cases, which I personally think we should stop doing when it becomes the default dtype.

+1

[simonjayhawkins]

StringDtype(storage="pyarrow", semantics="numpy")? or instead of semantics, could use "na_value=np.nan`

or maybe "nullable=[True|False]"

However, at the moment, we distinguish the nullable data types for the other dtypes (int, float, etc) with capitalization and so for consistency could also consider string/String as the dtypes.

[simonjayhawkins]

Suggested change

	dtype that has massive benefits for our users, both in usability as (for the
	significant part of the user base that has PyArrow installed) in performance.
	dtype that has massive benefits for our users, both in usability and, for users that already have PyArrow installed or have no issues installing PyArrow, in performance.

[simonjayhawkins]

a 2.3 release (maybe around the same time as 3.0rc) sounds reasonable.

If the features/bugfixes added to 2.3 are limited to the string dtype then we shouldn't need many patch releases. We may not need to fix any string dtype related issues that are fixed for 3.0 as these will be behind a flag in 2.3 and so shouldn't break existing code.

On the other hand, as these features are behind a flag, maybe releasing a 2.3 would not gain the field testing we hope for.

And therefore, instead of doing a 2.3, planning for at least a couple of release candidates for 3.0 would better achieve this.

[Dr-Irv]

Isn't the "in-house functionally-equivalent (but slower) version" the current implementation based on numpy 1.x in version 2.2, but we now make the dtype string instead of object ?

[jorisvandenbossche]

In the section below that expands on this object-type based implementation ("Object-dtype "fallback" implementation"), there is a bit longer explanation and I also link to the open PR implementing this: #58451

It is based on the current StringDtype / StringArray (using object dtype under the hood), and not directly on how object-dtype columns work right now. But anyway, both use the same implementation for the string accessor methods, and this new variant will also use that same implementation.

(and fwiw, this is not specific to numpy 1.x, it will also work on numpy 2.x, it just uses object dtype)

[Dr-Irv]

I have to admit that I am pretty confused by these various implementations and the corresponding semantics used to describe them.

I'd suggest having a summary of what exists today in pandas 2.2 (strings with object dtype, np.nan), the current StringDtype with pd.NA, the "experimental" pyarrow based implementations (with both pd.NA and np.nan being available), and anything else, and what is proposed would be available due to this PDEP, and how it might change in the future due to however we decide to handle missing values in the future, as well as nanoarrow and numpy 2.0 strings.

[jorisvandenbossche]

Current situation (pandas 2.2) is:

• Current default: object dtype with np.nan or None
• Experimental opt-in string dtypes using pd.NA: StringDtype() with storage being "python" (default, object dtype under the hood) or "pyarrow"
• Future string dtypes using NaN (behind pd.options.future.infer_string = True): StringDtype() with storage being "pyarrow_numpy"
  • This is the dtype that is essentially being proposed in this PDEP, but already exists since pandas 2.1 (#54792)

And then this PDEP also describes to add an extra option for the third bullet point, i.e. having a StringDtype() using NaN but backed by object-dtype array instead of pyarrow, which I dubbed (for now) "python_numpy" in the PR adding this (#58451), but that name is still being discussed.

As a starter (still thinking about how to make this clearer in the PDEP), does the above clarify it for you?

Note that the above listing leaves out pd.ArrowDtype(<some pyarrow string type>), because that is not really relevant for the discussion right now and I am not sure mentioning it will help, but that is yet another way to store strings.

[Dr-Irv]

As a starter (still thinking about how to make this clearer in the PDEP), does the above clarify it for you?

Yes, although I'd suggest for the PDEP a table that indicates the constructor version of the dtype (e.g., dtype=object, or dtype=pd.StringDtype()), and the string representation of the dtype.

Note that the above listing leaves out pd.ArrowDtype(<some pyarrow string type>), because that is not really relevant for the discussion right now and I am not sure mentioning it will help, but that is yet another way to store strings.

Is that equivalent to having a pure "pyarrow" backed string that uses pd.NA as the null semantics? If not, where does that fit it? Or is it not available?

[jorisvandenbossche]

Is that equivalent to having a pure "pyarrow" backed string that uses pd.NA as the null semantics? If not, where does that fit it? Or is it not available?

StringDtype(storage="pyarrow") is also a "pure pyarrow backed string that uses NA", to be clear. But so
StringDtype(storage="pyarrow")and ArrowDtype(pa.(large_)string()) are essentially equivalent, except for using a different dtype and array class.

[Dr-Irv]

Don't you mean "numpy 2.0 string-dtype" ? Because the "numpy object-dtype" is currently there? This labeling is confusing.

Given the discussions elsewhere about where the names are now for the dtype "string[pyarrow]", "string[pyarrow_numpy]", etc. which I can't keep track of, I think that the nomenclature in terms of strings that work should be specified, comparing what is in 2.2 to what would be implemented as a result of this PDEP.

The possible strings are confusing. Which strings can be used in a dtype argument in a constructor or astype() ? Which strings would be seen by users when they do Series.dtype ? There is what currently exists in pandas 2.2, and what would exist based on this PDEP, and I'm not seeing what the conclusion of that would be.

[jorisvandenbossche]

Because the "numpy object-dtype" is currently there?

No, it is not yet there (see my answer to your previous comment)

[WillAyd]

PDEP-13 proposes StringDtype(backend="pyarrow", na_marker=np.nan). I think the repr should just be updated to reflect that; trying to sift through the meaning of int versus Int versus int[pyarrow] compared to string versus string[pyarrow] versus string[pyarrow_numpy] I think would be a distraction for this proposal

[WillAyd]

I am not sure this argument is that convincing either, although for slightly different reasons. I don't think we need to feel rushed for the next release

[WillAyd]

the challenges around this will not be unique to the string dtype and
therefore not a reason to delay this.

I might be missing the intent but I don't understand why the larger issue of NA handling means we should be faster to implement this

[jorisvandenbossche]

I don't understand why the larger issue of NA handling means we should be faster to implement this

It's not a reason to do it "faster", but I meant to say that the discussion regarding NA is not a reason to do it "slower" (to delay introducing a dedicated string dtype)

[WillAyd]

I think the flip side is that if we aren't careful about the NA handling we can introduce some new keywords / terminology that makes it very confusing in the long run (which is essentially one of the problems with our strings naming conventions)

As a practical example, if we decided we wanted semantics= as a keyword argument to StringDtype in this PDEP to move the NA discussion along, that might be counter-productive when we look at more data types and decide semantics= was not a clear way to allow datetime data types to support pd.NaT as the missing value.

(not saying the above is necessarily the truth, just cherry picking from conversation so far)

[jorisvandenbossche]

That's one reason that I personally would prefer not introducing a keyword specifically for the missing value semantics, for now (just for this PDEP / the string dtype). I just listed some options in #58613, and I think we can do without it.

[WillAyd]

This just retroactively clarifies the reasoning for string[pyarrow_numpy] to have existed in the first place right? Or is it supposed to be hinting at some other feature that the implementation details of the PDEP is proposing?

[jorisvandenbossche]

Yes, it's indeed explaining why we did this, which is of course "retroactively" given I was asked to write this PDEP partly for changes that have already been released. So a big part of the PDEP is retroactively in that sense (which it not necessarily helping to write it clearly ..).

[simonjayhawkins]

Or is it supposed to be hinting at some other feature that the implementation details of the PDEP is proposing?

however, more importantly, the PDEP makes this (the already added dtype) the default in 3.0. It would remain behind the future flag for the next release if enough people feel we are not ready.

[WillAyd]

Historically you would get this by using dtype="string" too right? I'm a little wary that we are underestimating the scope of how breaking this could be; I didn't even realize we considered that dtype experimental all this time

[simonjayhawkins]

This has been available (as pyarrow backed) since 1.3, so almost three years (July 2, 2021). Even though considered experimental, if the new string dtype is not accepted for 3.0, then maybe a deprecation warning should be added? (We could also do this if decided a 2.3 release is needed?)

[jorisvandenbossche]

A deprecation warning about what exactly?

[jorisvandenbossche]

I'm a little wary that we are underestimating the scope of how breaking this could be

The scope of changing NaN to NA for all users is much bigger though (essentially what was decided in PDEP-10 if we would follow it strictly to the letter).
And similarly if we would in the future change NaN/NaT semantics to NA for all dtypes, the scope will be much bigger (because once that is enabled by default, for example a user that was doing dtype="float64" will probably get the new NA behaviour while now it uses NaN), but we are still considering that (granted, it's exactly those details that we have to discuss a lot more in detail (elsewhere) and figure out, though).

I know that this is not necessarily a good argument to justify this breaking change (because we certainly should be wary of the scope of those breaking changes), but I do want to point out again that the choice in this PDEP to use NaN semantics is to reduce the scope of the breaking changes for most users (at the expense of increasing the scope of breaking changes for the smaller subset of users that was already using dtype="string").

If we don't want to make dtype="string" breaking, then either we need to come up with a different name for the dtype (not using "string", like "utf8" or "text"), or either we need to delay introducing a default string dtype until after we have agreement on the NA discussions.

And personally I think "string" is by far the best name (and I find the small breakage worth it for being able to use that name), and as I argued elsewhere (and in the Why not delay introducing a default string dtype? section in the PDEP text), I think it is valuable for our users to not wait with adding a dedicated string dtype until we are ready with the NA discussion and implementation.

[WillAyd]

at the expense of increasing the scope of breaking changes for the smaller subset of users that was already using dtype="string"

This is where I am a little uncomfortable - I don't know how to measure the size of that, but I am wary of assuming it is not a signifcant number of users. The fact that "string" returns NA as a missing value is a documented difference in our code base:

https://pandas.pydata.org/docs/dev/user_guide/text.html#behavior-differences

And its usage has been promoted for quite some time:

https://stackoverflow.com/a/60553529/621736
https://towardsdatascience.com/why-we-need-to-use-pandas-new-string-dtype-instead-of-object-for-textual-data-6fd419842e24
https://pandas.pydata.org/pandas-docs/stable/whatsnew/v1.1.0.html#all-dtypes-can-now-be-converted-to-stringdtype

If we don't want to make dtype="string" breaking, then either we need to come up with a different name for the dtype (not using "string", like "utf8" or "text"), or either we need to delay introducing a default string dtype until after we have agreement on the NA discussions.

Yea none of these options are great...but out of them I still would probably prefer waiting. I think right now we are marching down a path of "string" missing values:

1. Returning pd.NA today
2. Returning np.nan with this PDEP (granted those changes are already in main)
3. Going back to returning pd.NA with the NA PDEP

[Dr-Irv]

But personally I think dtype="string" meaning something different than the default string dtype you get without specifying the dtype is going to be very confusing ..)

I think we have to carefully specify what the user specifies in a dtype argument and how that gets interpreted, versus what we return as the dtype when they look at Series.dtype.

So we could have a mapping that says

User specifies dtype=	pandas returns Series.dtype
Unspecified	"string[pyarrow_numpy]" OR "string[python]"
"string"	"string[pyarrow]"
StringDtype("pyarrow")	"string[pyarrow]"
StringDtype("python")	"string[python]"
StringDtype("pyarrow_numpy")	"string[pyarrow_numpy]"

The first row depends on whether pyarrow is installed.
For the second, third and fifth rows, if pyarrow is not installed, we raise an Exception.

Separately, we can then debate what the values in the second column should look like in #58613 . I personally am not a fan of "pyarrow_numpy"

[WillAyd]

No, my answer to your example snippet was trying to explain how I would ensure this does not break (if we return bool column instead of object dtype with True/False/NaN will ensure that filtering keeps working).

Ah OK - I didn't realize you were proposing that change be a part of this PDEP, just thought it was an idea you had for the future. But that's a completely new behavior...and then begs the question of do we go back and change dtype=object to have that same behavior or just have dtype="string" exclusively have it. Ultimately we end up with the same issue

[lithomas1]

Yeah, I also agree with Will that it's not fair to change this without warning for people already using "string".
(pd.NA is also a big selling point of the dtype="string" too)

Maybe a good compromise would be to use string[pyarrow] under the hood for those users (if they had it installed)?

If we were to move ahead with the move to nullable dtypes in general, I worry that this changing of the na value for dtype="string" from pd.NA -> np.nan -> pd.NA will cause a lot of confusion.

If we were to do 2.3 (like I suggested below), this might be addressable there (with a deprecation).

[jorisvandenbossche]

Still adding some deprecation warnings in 2.x for current users of StringDtype is something we certainly could do. I am personally ambivalent about it, but fine with adding it if others think that is better (I do think it might become quite noisy, and it also does not change the fact that 3.0 would switch from NA to NaN)

The warning message could then point people to enable pd.options.future.infer_string = True in case they only care about having the (faster) string dtype, or otherwise update their dtype specification if they want the NA instead of NaN version.

[jorisvandenbossche]

I think we have to carefully specify what the user specifies in a dtype argument and how that gets interpreted, versus what we return as the dtype when they look at Series.dtype.

So we could have a mapping that says

I created a variant of that table #58613 (comment) with a concrete proposal

For the second, third and fifth rows, if pyarrow is not installed, we raise an Exception.

(for clarity, this "second" row referred to specifying a dtype with "string")
If you explicitly ask for pyarrow, then yes raising an exception is fine and expected. But a generic "string" (or StringDtype()) has to mean "whatever string dtype that is the default" and so cannot raise an exception if pyarrow is not installed, but should return the object-dtype based fallback.

[jorisvandenbossche]

One of the concrete discussion points is the API design of the StringDtype(..) constructor and the way to distinguish the various variants of the dtype (i.e. the current "pyarrow_numpy" naming we introduced in #54533 / #54792).
To keep that sub-discussion manageable, I opened a dedicated issue for that specific topic: #58613

[jbrockmendel]

I'm with Joris pretty much across the board on this. I'm pretty sure @phofl will be too.

[jorisvandenbossche]

Thanks Brock. It would indeed be good to hear from others that previously seemed to be OK with the compromise and the NaN behaviour we currently have on main (or not OK, of course, in that case you are also allowed to speak up ;))

[jorisvandenbossche]

@pandas-dev/pandas-core I pushed a set of updates based on the discussions from last week:

• Expanded the section on "Missing value semantics" to more clearly contrast the behaviour differences between what is being proposed and the existing StringDtype (as it seems that keeps causing some confusion)
• Updated the naming to use StringDtype() with a combo of storage and na_value keywords (so replacing the confusing storage="pyarrow_numpy" name with storage="pyarrow", na_value=pd.NA). This is based on the proposal I did in Default string dtype (PDEP-14): naming convention to distinguish the dtype variants #58613 (comment), we can keep discussing that aspect there.
• Expanded the backwards compatibility section to more explicitly call out the valid backwards compatibility issues for existing users of dtype="string"/dtype=pd.StringDtype(). Although I know this is one of the contentious parts, I am still proposing to introduce breaking changes for those users (I don't think there is any way around this given the proposal of using NaN semantics for the default string dtype), but based on the feedback I added the proposal to add a deprecation warning for this in advance. So at least it's a breaking change for which we warn in advance and for which we provide am easy option to suppress the warning / preserve the current behaviour with minimal code edits required.

My understanding is that the main points of contention currently are:

• The notion that the default string dtype should use NaN semantics to be consistent with the other default dtypes, as proposed by the PDEP (and thus also return default numeric/bool dtypes in string operations, in contrast to the nullable/masked Integer/Float/Boolean dtype being returned by the current StringDtype variant that uses pd.NA).
• The fact that letting the implicit dtype="string" / dtype=pd.StringDtype() become an alias for the new default string dtype (NaN-variant), while it currently already gives the NA-variant, means this part is a breaking change for the existing users of StringDtype (although we can add a deprecation warning for it in advance, it's of course still a change in behaviour in 3.0 for those users)
• Related to the above points, but the notion that we should introduce this change (a default string dtype) now for pandas 3.0, instead of waiting for pandas 4.0 when we (hopefully) have a better idea about using pd.NA more generally / about a general logical dtype system (instead of already doing something like that but just for the string dtype), which could avoid both points above.

While I firmly believe for the first bullet point that this is the only viable option at this point (IMO we really don't want to give users mixed NaN/NA semantics as their default user experience), I think those points are further mostly subjective judgement calls about whether the added complexity (having yet another variant of StringDtype to be able to make this a default dtype right now) and breaking changes (for existing users of StringDtype) are worth it compared to the added benefit of having a dedicated (and potentially much faster) string dtype by default sooner rather than later.

[Dr-Irv]

Most of my changes are about changing the grammar from first person to third person. A few other comments are questions related to clarifying behavior. Otherwise, I think I'm getting closer to agreeing to this.

[Dr-Irv]

For clarification, isn't the alternative not the "numpy object-dtype alternative", but rather an extension array using numpy objects as strings, with np.nan missing value semantics. You're not proposing that you still get a numpy backed array with object dtype, right?

[jorisvandenbossche]

You're not proposing that you still get a numpy backed array with object dtype, right?

Right, definitely not proposing that. I meant the alternative ExtensionArray using numpy object-dtype under the hood. Will need to clarify that.

[jorisvandenbossche]

@Dr-Irv thanks for the detailed review! Merged most of the suggestions, and answering the remaining comments

[WillAyd]

I'm still -1 on changing this behavior; I do not want to revert "string" back to NumPy nullability semantics; that is a breaking change for anyone that has been using our extension type system to "solve" this issue for the past 5-6 years

[WillAyd]

I think the PDEP should also be clear about long term expectations. I still think right now we are assuming:

• 2.x release - dtype="string" uses pd.NA as a missing value marker
• 3.x release - dtype="string" uses np.nan as a missing value marker by default, user setting to change to pd.NA
• 4.x release - dtype="string" changes back to the 2.x behavior

[jorisvandenbossche]

I'm still -1 on changing this behavior; I do not want to revert "string" back to NumPy nullability semantics

For clarification, do you mean "-1 on using NaN semantics for the default string dtype, regardless of how we name it", or only "-1 on using NaN semantics for the dtype created as dtype="string"" ?

Because it is only the latter that causes the breaking change for anyone already using the nullable string dtype. Assume we would use a different name or different string alias than "string", we could still have a default string dtype (which everyone that was not yet using the nullable StringDtype would get by default) that uses the proposed NaN semantics, while not causing a breaking change for the existing users of dtype="string" / dtype=pd.StringDtype().

(it's another question whether there is enough support for using a different name, I personally think "string" is the best choice which we should reserve for the default dtype, but first to get a good understanding of your position)

[WillAyd]

For clarification, do you mean "-1 on using NaN semantics for the default string dtype, regardless of how we name it", or only "-1 on using NaN semantics for the dtype created as dtype="string"" ?

Definitely the latter, maybe the former. My expectation with PDEP-10 was that the default pyarrow string would be using pd.NA. If that is too difficult then yea there probably is a compromise on the former, but I do not want to take away the dtype="string" functionality from users that has been working all of this time.

Not that it is ideal, but we already have dtype=str today and dtype="string"; maybe the former becomes the new name for what is being proposed here instead of string[pyarrow_numpy] and only change dtype="string" to be pyarrow backed without changing nullability semantics?

That doesn't solve the str/"string" discrepancy but I don't think introduces any new problems either

[jorisvandenbossche]

I do not want to take away the dtype="string" functionality from users that has been working all of this time.

We are not "taking away" that functionality, the current revision of the PDEP only asks users to use dtype=pd.StringDtype(na_value=pd.NA) instead to continue using the same functionality, and Irv's suggestion would minimize the required code change to use dtype="String"

Not that it is ideal, but we already have dtype=str today and dtype="string"; maybe the former becomes the new name for what is being proposed here instead of string[pyarrow_numpy]

Then what would you propose to show in the df.dtypes output? (i.e. the string repr of the dtype) Also "str" instead of "string"?
That would be an option. In that case, we could also use a separate StrDtype() class for those NaN-variants (which also solves the back compat issue for dtype=pd.StringDtype()).

[Dr-Irv]

Interesting idea, but I think I'm -0.5 on doing that. The proposal here is more in line with PDEP-13's notion of logical types and can segue into that. I think having pd.StrDtype() and pd.StringDtype() only moves us away from that.

I'm not sure I agree with you here @WillAyd . If we had both, and we do logical dtypes, then we pick one or the other. Similar to how we have to reconcile dtype=int versus dtype=pd.Int64Dtype() in the future if we use logical types, which would also need to reconcile the NaN vs pd.NA issue.

[WillAyd]

The idea is that the logical type is decoupled from null handling. If you ignored missing values for a minute, what value do you think there is in having a separate Str and String type in a type system?

[Dr-Irv]

The idea is that the logical type is decoupled from null handling. If you ignored missing values for a minute, what value do you think there is in having a separate Str and String type in a type system?

No value in a type system. I just think it may be useful to have this now so we can manage the transition from where we are today to 3.0, and from 3.0 to a future state.

[WillAyd]

This is something we can discuss further in PDEP-13 but the current revision of it proposes consistently having the format of <TYPE>Dtype(na_marker=pd.NA|"legacy"), the idea being that long term pd.NA can be used consistently, but we will have a compatability period of "legacy" where you get the mix of np.nan / pd.NaT for NumPy-based types (and still probably pd.NA for any new types like ListDtype).

So StringDtype(na_marker=np.nan) is slightly different from that. Maybe asking users to explicitly say np.nan instead of "legacy" has some downsides from a UI perspective, but my gut feeling is that we can solve that over time

[WillAyd]

I do want to be wary though of users expecting complete control over the na_marker field. I don't see a value-add in trying to support DatetimeDtype(na_marker=np.nan) alongside DatetimeDtype(na_marker=pd.NaT) nor do I think there would ever be value in ListDtype(na_marker=np.nan)

[WillAyd]

I don't have an alternate proposal but I think our naming is even more confusing with this when using the dtype_backend="numpy_nullable" argument in I/O.

I assume that will yield pd.StringDtype("pyarrow", na_marker=pd.NA) or pd.StringDtype("python", na_marker=pd.NA) in this proposal but never pd.StringDtype(..., na_marker=np.nan), so even though it is numpy_nullable it never uses NumPy nullability

[jorisvandenbossche]

so even though it is numpy_nullable it never uses NumPy nullability

I am not going to argue this is a good name (I never liked this), but this option right now does mean "give me dtypes with NA semantics (implemented using numpy under the hood, instead of using pyarrow)", and so yes it will give you the NA-variant of the dtype.

This option does not mean to ask for Numpy-like (NaN) nullability semantics. That can be confusing, but then that is the case in general, and not related to just the string dtype.

[simonjayhawkins]

If one was to argue that the users that benefit most from a dedicated string dtype are already aware of the "experimental" string dtype that has been pyarrow backed for almost 3 years, then one could also argue that there is probably not the benefits to users and urgency that this PDEP initially proposed. (Only a breaking change for those that do already benefit)

Also, given the evolution of this discussion and updates to this PDEP, I think that my comment in #57073 about not being ready has probably gained some traction since this proposal now suggests further releases before 3.0.

However, I was also under the impression that the intention of the solution as initially proposed was to help get the 3.0 released unblocked if the PyArrow dependency requirement was dropped. If we approve this PDEP with the modifications and that results in pandas 3.0 being released much later than planned, we all the time move closer to the point where the NumPy native string solution may become a usable solution (and use the time to enhance the pandas I/O and 2d EA interface to better support it)

Assuming that I could safely say that nobody really likes any fallback solution for either performance, consistency, complexity, confusion or maintenance reasons then we should probably include in this PDEP the deprecation plan for the fallback. As I mentioned in #57073 (comment), which is expected to happen first, having PyArrow as a required dependency or having the minimum version of NumPy as 2.0?

[Dr-Irv]

Found a few other first-person/third-person things and some backticks

[phofl]

I'm with Joris pretty much across the board on this. I'm pretty sure @phofl will be too.

Agreed with @jbrockmendel

This would be an acceptable compromise of not requiring PyArrow to me

[rhshadrach]

I'm on board with this proposal. Only minor issue is below. The issue was a misunderstanding by me.

[MarcoGorelli]

On board with the idea!

Making a nullable dtype available by default would require a lot more discussion which won't happen in time for 3.0 (which I'd also rather not delay too much). In particular, I know some people here who feel quite strongly (but with opposite opinions) about the NaN vs null topic, and that'll require further discussion

Happy to discuss nullable dtypes by default for the 2025 major release. Though in particular I'm curious about your (Joris') thoughts on whether you'd eventually be happy making PyArrow required if PyArrow dtypes were the default wherever possible

I see Will's point about making changes to StringDType, although given its experimental status, I think it'd be OK. And I also like Irv's 'String' suggestion, which feels consistent with 'int64' vs 'Int64'

Thanks for all the effort you put into this proposal and to answering so many questions

[MarcoGorelli]

I'd include "feedback from maintainers of other packages" too