cmake: picky-linker fixes for openssl, ZLIB, H3 and more

[vszakats]

• fix HTTP/3 support detection with OpenSSL/quictls built with ZLIB.
  (Requires curl be built with ZLIB option also.)

• fix HTTP/3 support detection with OpenSSL/quictls/LibreSSL and ld
  linker on Windows.

• fix HTTP/3 support detection with wolfSSL to automatically add
  ws2_32 to the lib list on Windows. For all linkers.

• reposition ZLIB (and other compression) detection after TLS
  detection, but before calling HTTP/3-support detection via
  CheckQuicSupportInOpenSSL.

  May be a regression from ebef55a
  May fix undefined reference for zlib symbols when building with static dependencies and BUILD_SHARED_LIBS=OFF #10832 (Reported-by: Micah Snyder)

  This also seems to fix an odd case, where OpenSSL/quictls is correctly
  detected, but its header path is not set while compiling, breaking
  build at src/curl_ntlm_core.c. Reason for this remains undiscovered.

• satisfy "picky" linkers such as ld with MinGW, that are highly
  sensitive to lib order, by also adding brotli to the beginning of the
  lib list.

• satisfy "picky" linkers by adding certain Windows systems libs to
  the lib list for OpenSSL/LibreSSL. (Might need additional ones for
  other forks, such as pthread for BoringSSL.)

Note: It'd make sense to always add ws2_32, crypt32 (except
Windows App targets perhaps?), bcrypt (except old-mingw!) on Windows
at this point. They are almost always required, and if some aren't,
they are ignored by the linker with no effect on final binaries.

Closes #10857

---

With these, a CMake build with gcc/ld almost works ;) Now gsasl lib is in the wrong position ¯\(ツ)/¯. → Worked around using CMAKE_C_STANDARD_LIBRARIES to pass it.

[jay]

not against this but should we be making concessions for these users that are compiling static builds? things can always break as the static library changes dependencies. for example what if the reporter did not enable zlib for curl, then isn't the problem the same, his build will fail because his ssl still requires zlib but cmake does not know that?

shouldn't the burden be on him to specify the requirements if there's no PKG_CONFIG="pkg-config --static" or other autotools way that cmake can't use. even then something like nghttp2 at least on windows you have to specify CPPFLAGS="-DNGHTTP2_STATICLIB" as well. it seems static builds are a lot of trouble this way.

[vszakats]

@jay: It'd definitely help to state that. Static builds are extremely fragile and labour-intensive, curl-for-win being the living proof for that. An infinite amount of time can be spent tracking the variations and their changes. E.g. BoringSSL still needs phtread and indeed, those extra macros here and there. Also, CMake and autotools are actively working against making static linking work.

On Windows, I think it'd also help if we added the base set of Windows system libs that are almost always required. They are a common source of problems there.

[bagder]

I am in favor of clearly documenting that doing working completely static curl builds is not the responsibility of the curl build scripts. To set the expectations right.

[vszakats]

Here's one reply (and some more below that), to a static linking report/question received earlier: curl/curl-for-win#39 (comment) (This was Windows-specific, which is a particularly problematic platform for this, but some of it is likely common to all.)

And an unresolved discussion with a similar request: curl/curl-for-win#41

These were for final apps linking libcurl, which is kind of the same as linking curl itself.

[jay]

in mingw-w64 I used to do it like below for autotools. i don't know if something similar is possible with cmake.

./buildconf
CPPFLAGS="-DNGHTTP2_STATICLIB" LDFLAGS="-static" PKG_CONFIG="pkg-config --static" ./configure ...
make LDFLAGS="-static -all-static" V=1

then tests

make test LDFLAGS="-static -all-static" V=1

then i would actually run the tests from msys2 shell instead of mingw-w64 shell for some reason. it's been a while since i've built like that.

[vszakats]

@jay: Thanks, I did not know about the -all-static option. Maybe it can help dropping some hacks from curl-for-win autotools builds. The MSYS2 env helps a lot indeed. Some believe that the resulting binary also requires MSYS2, but that is not the case (with -static-libgcc / -static-libstdc++) binaries.

[dg0yt]

I doubt it ever made sense to PREPEND in addition to APPEND.

[vszakats]

This has been a while ago, and can't recall the exact test results for this specific case. But, it's unlikely I'd add such thing unless required to fix the picky-linker problem. (A clean solution would be -Wl,--start-group / -Wl,--end-group, but could not figure out how to apply that to CMake.)

If a better solution to this exists, I'd be very glad to learn about it. It's been a two-decade problem with this linker and static linking.

[dg0yt]

The "picky-linker problem" is unknown to me, but I know that the order of static libs matters outside MSVC.
And I know that CMake may mess with the order unless it is properly defined by target dependencies. Subject to CMake version.
(And I know that meson still sorts link libs it gets from CMake...)

FTR I'm about to patch out the PREPEND line in the vcpkg port. Vcpkg CI builds with static library linkage for windows, linux, osx, and android. And of course I'm just tracing a linking issue. I want to get rid of distractors.

[vszakats]

"Picky-linker" is my term, it applies to binutils ld specifically. It may apply to others, but I haven't seen it anywhere else. Also llvm/clang's lld doesn't have this problem, neither does MSVC. My understanding is that the linker does no effort to resolve symbols in a second pass, so if a symbol is referenced earlier than defined, linking breaks. It must have a strong reason doing this for decades and without a fix.

Feel free to patch it out locally, though I'm not aware of any issues it may cause. You may also move it inside an if(CMAKE_COMPILER_IS_GNUCC), if it does.

[dg0yt]

My understanding is that the linker does no effort to resolve symbols in a second pass, so if a symbol is referenced earlier than defined, linking breaks.

It is the other way round. First the dependent lib (-lbrotlidec), then the independent lib (-lbrotlicommon).
If you get it wrong, then you need to repeat it.

And that's why extra prepending is still in the "never made sense" box for me.
(Hint: -lz and -lm are typically found at the end of the link libs.)

And that's why "sort" and "remove duplicates" do not work on link libs.

[vszakats]

Would this help with the root cause for brotli?:

diff --git a/CMake/FindBrotli.cmake b/CMake/FindBrotli.cmake
index 11ab7f825..dcab8f7cd 100644
--- a/CMake/FindBrotli.cmake
+++ b/CMake/FindBrotli.cmake
@@ -40,4 +40,4 @@ find_package_handle_standard_args(Brotli
 )
 
 set(BROTLI_INCLUDE_DIRS ${BROTLI_INCLUDE_DIR})
-set(BROTLI_LIBRARIES ${BROTLICOMMON_LIBRARY} ${BROTLIDEC_LIBRARY})
+set(BROTLI_LIBRARIES ${BROTLIDEC_LIBRARY} ${BROTLICOMMON_LIBRARY})

I'm trying to replicate it, but without knowing the exact trigger configuration, this might take some time. If it triggers in my current test setup at all.

edit: managed to replicate and confirm that the above fix works, without the prepend hack.