build: upgrade airflow to 3.1.3

[Lee-W]

Types of changes

• Other (please describe)

Description

Update apache-airflow to 3.1.3. Manually patched apache/airflow#58130 and apache/airflow#58229 which are likely to be included in 3.1.4

Checklist

• Add test cases to all the changes you introduce
• Run make lint and make test locally to ensure all linter checks and testing pass
• Update the documentation if necessary

Steps to Test This Pull Request

Expected behavior

Related Issue

Additional context

[Lee-W]

Hmmm... I guess I'll need a way to check the consistency of this

[Lee-W]

This does not yet work. Might need some SQLite migration fix. It breaks in 0082_3_1_0_make_bundle_name_not_nullable.py. Looks like the same FK issue we previously encountered.

 The above exception was the direct cause of the following exception:

 Traceback (most recent call last):
File "/app/.venv/bin/airflow", line 10, in <module>
  sys.exit(main())
File "/.../python3.10/site-packages/airflow/__main__.py", line 55, in main
  args.func(args)
File "/.../python3.10/site-packages/airflow/cli/cli_config.py", line 49, in command
  return func(*args, **kwargs)
File "/.../python3.10/site-packages/airflow/utils/cli.py", line 114, in wrapper
  return f(*args, **kwargs)
File "/.../python3.10/site-packages/airflow/utils/providers_configuration_loader.py", line 54, in wrapped_function
  return func(*args, **kwargs)
File "/.../python3.10/site-packages/airflow/cli/commands/db_command.py", line 206, in migratedb
  run_db_migrate_command(args, db.upgradedb, _REVISION_HEADS_MAP)
File "/.../python3.10/site-packages/airflow/cli/commands/db_command.py", line 134, in run_db_migrate_command
  command(
File "/.../python3.10/site-packages/airflow/utils/session.py", line 100, in wrapper
  return func(*args, session=session, **kwargs)
File "/.../python3.10/site-packages/airflow/utils/db.py", line 1136, in upgradedb
  command.upgrade(config, revision=to_revision or "heads")
File "/.../python3.10/site-packages/alembic/command.py", line 483, in upgrade
  script.run_env()
File "/.../python3.10/site-packages/alembic/script/base.py", line 545, in run_env
  util.load_python_file(self.dir, "env.py")
File "/.../python3.10/site-packages/alembic/util/pyfiles.py", line 116, in load_python_file
  module = load_module_py(module_id, path)
File "/.../python3.10/site-packages/alembic/util/pyfiles.py", line 136, in load_module_py
  spec.loader.exec_module(module)  # type: ignore
File "<frozen importlib._bootstrap_external>", line 883, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/.../python3.10/site-packages/airflow/migrations/env.py", line 138, in <module>
  run_migrations_online()
File "/.../python3.10/site-packages/airflow/migrations/env.py", line 132, in run_migrations_online
  context.run_migrations()
File "<string>", line 8, in run_migrations
File "/.../python3.10/site-packages/alembic/runtime/environment.py", line 946, in run_migrations
  self.get_context().run_migrations(**kw)
File "/.../python3.10/site-packages/alembic/runtime/migration.py", line 627, in run_migrations
  step.migration_fn(**kw)
File "/.../python3.10/site-packages/airflow/migrations/versions/0082_3_1_0_make_bundle_name_not_nullable.py", line 73, in upgrade
  with op.batch_alter_table("dag", schema=None) as batch_op:
File "/usr/python/lib/python3.10/contextlib.py", line 142, in __exit__
  next(self.gen)
File "/.../python3.10/site-packages/alembic/operations/base.py", line 397, in batch_alter_table
  impl.flush()
File "/.../python3.10/site-packages/alembic/operations/batch.py", line 163, in flush
  batch_impl._create(self.impl)
File "/.../python3.10/site-packages/alembic/operations/batch.py", line 468, in _create
  op_impl.drop_table(self.table)
File "/.../python3.10/site-packages/alembic/ddl/impl.py", line 446, in drop_table
  self._exec(schema.DropTable(table, **kw))
File "/.../python3.10/site-packages/alembic/ddl/impl.py", line 246, in _exec
  return conn.execute(construct, params)
File "/.../python3.10/site-packages/sqlalchemy/future/engine.py", line 286, in execute
  return self._execute_20(
File "/.../python3.10/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
  return meth(self, args_10style, kwargs_10style, execution_options)
File "/.../python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
  return connection._execute_ddl(
File "/.../python3.10/site-packages/sqlalchemy/engine/base.py", line 1477, in _execute_ddl
  ret = self._execute_context(
File "/.../python3.10/site-packages/sqlalchemy/engine/base.py", line 1953, in _execute_context
  self._handle_dbapi_exception(
File "/.../python3.10/site-packages/sqlalchemy/engine/base.py", line 2134, in _handle_dbapi_exception
  util.raise_(
File "/.../python3.10/site-packages/sqlalchemy/util/compat.py", line 211, in raise_
  raise exception
File "/.../python3.10/site-packages/sqlalchemy/engine/base.py", line 1910, in _execute_context
  self.dialect.do_execute(
File "/.../python3.10/site-packages/sqlalchemy/engine/default.py", line 736, in do_execute
  cursor.execute(statement, parameters)
 sqlalchemy.exc.IntegrityError: (sqlite3.IntegrityError) FOREIGN KEY constraint failed
 [SQL:
 DROP TABLE dag]
 (Background on this error at: https://sqlalche.me/e/14/gkpj)