Duplicate references in POT files and warnings about redefining modules

[barttenbrinke]

When I run
mix gettext.extract --merge

on my umrella project, I get a LOT of warnings saying:

warning: redefining module Core.Reports.ReportThree.Report (current version loaded from /_build/dev/lib/core/ebin/Elixir.Core.Reports.ReportThree.Report.beam)
  lib/core/reports/report_three.ex:24

Even if I delete _build - I think this is also causing this issue, on all of my POT & PO files, I get

#: lib/core/reports/report_three.ex:170
#: lib/core/reports/report_three.ex:170
#, elixir-autogen, elixir-format
msgid "Error"
msgstr "Error"

Which I assume is caused by the module being in memory twice or something, duplicating the reference.
I now just uniq the generated PO and POT files as a workaround, but I'd rather understand what the core issue is here.

[arjan]

Same issue here, now using a script like the following:

#! /usr/bin/env elixir
#
# fix-gettext <dir>
#
# Will dedup each file in the given dir, recursively.

System.argv()
|> Stream.flat_map(fn path ->
  if File.dir?(path) do
    Path.wildcard(Path.join(path, "**"))
  else
    path
  end
end)
|> Stream.reject(&File.dir?/1)
|> Stream.map(fn path ->
  data =
    path
    |> File.stream!()
    |> Enum.dedup()

  File.write!(path, data)

  IO.puts("Wrote to: #{path}")
end)
|> Stream.run()

[maennchen]

I believe this is caused by the same as #178

[maennchen]

@josevalim / @whatyouhide Would you potentially be open to a new extract methodology that should resolve the umbrella issue?

• Add config value extraction_environments, which defaults to [:dev]
• Add config value extraction_apps, which defaults to the current application
• The gettext macros will register a module attribute containing all translations and register an @before_compile, which exposes the translations with an __translations__/0 function if the current env is contained in extraction_environments.
• The Gettext.Extractor will read all modules translations from __translations__/0

Benefits of this approach:

• mix gettext.extract does not have to recompile since all translations are already available from the normal compilation
• This approach should easily work for umbrella applications or potentially even dependencies with a few tweaks

If you're open to this approach, I do have some time to try it and see how it performs.

[whatyouhide]

@maennchen that's a bigger change than I'd like. I’m ok to do it if it's the only way, but I wanna try to figure out if there is a simpler way instead. Have you tried playing a bit with the recompilation that is triggered by mix gettext.merge? Because if we recompile from scratch correctly we shouldn't be seeing these issues. 😕

[maennchen]

@whatyouhide I played around with it for a while in this project: https://github.com/jshmrtn/hygeia/tree/main

But even with a complete recompilation, I would run into multiple outcomes at random:

• New Translations not added
• Old Translations not removed
• Or all good

I used this mix tak, which made it better, but didn't resolve it completely.

https://github.com/jshmrtn/hygeia/blob/e3ac39437c19041609162d70baf4ff6e96f846ea/apps/hygeia_gettext/lib/mix/tasks/gettext/extract/umbrella.ex

I haven't really been able to pinpoint the source of the issue.

Everything was resolved when I merged all apps.

[PJUllrich]

Sorry, I don't have a solution either, but I also experience the flood of redefining module warnings when I run:

mix gettext.extract --check-up-to-date

Edit: I don't have an umbrella app though but I still see the same errors.

[maennchen]

@PJUllrich Do you by any chance have a way to reproduce this?

I only experienced this issue with umbrella applications and haven't seen the problem since I stopped using those.

[rhcarvalho]

I think I'm running into this and managed to reproduce on a brand-new Phoenix app.

Steps:

1. mix phx.new rachel --database sqlite3 --adapter cowboy --install
2. Add alias to mix.exs:

  def cli do
    [
      preferred_envs: [precommit: :test, ci: :test]
    ]
  end

  defp aliases do
    [
      # ...
      precommit: ["compile --warning-as-errors", "deps.unlock --unused", "format", "test"],
      ci: [
        "setup",
        "deps.unlock --check-unused",
        "format --check-formatted",
        # "gettext.extract --check-up-to-date",
        "test"
      ]
    ]
  end

3. Run mix ci multiple times, no "warning: redefining module" printed
4. Uncomment line gettext.extract --check-up-to-date
5. Run mix ci, prints "warning: redefining module" for every module

This is specially annoying as it conflicts with compile --warnings-as-errors, which I'm trying to enable globally, either by aliasing the compile task or with elixirc_options: [warnings_as_errors: true] in MixProject.project/0 (update: this is actually deprecated).

---

Versions:

• Phoenix 1.8.1
• Elixir 1.18.4
• Erlang 27.3.4.2

[rhcarvalho]

The solution I found so far to have both compile --warnings-as-errors and gettext.extract --check-up-to-date is to run them on separate processes using cmd mix ...:

# mix.exs

  defp aliases do
    [
      compile: "compile --warnings-as-errors",
      test: "test --warnings-as-errors",
      # ...
      ci: [
        "setup",
        "deps.unlock --check-unused",
        "format --check-formatted",
        "cmd mix gettext.extract --check-up-to-date",    # <---
        "test"
      ]
    ]
  end