Out of memory after thousands of optimization tasks

[lidingxu]

I optimize thousands of tiny LPs using Gurobi, but JuMP outputs an error after several iterations: "exception outside of a @test Gurobi Error 10001". My memory runs out soon. The code is straightforward:
`

for i in collect(1:10000)

    model = Model(Gurobi.Optimizer)

    set_optimizer_attribute(model, "Threads", option.thread)
    set_optimizer_attribute(model, "OutputFlag", option.log_level)
    set_optimizer_attribute(model, "TimeLimit", 10) # note Gurobi only supports wall-clock time
    set_silent(model)
    # variables
    @variables(model, begin 
        0 <= x <= 1 # 
        0 <= y <= 0.2 # 
    end) 
    # constraints
    @constraints(model, begin 
        x + y <=1
    end) 
    # objective
    @objective(model, Min, 0)
    optimize!(model)   
end

`

It seems that the backend solver's memory is not released per iteration. Does anyone know how to fix this problem? Thanks.

The error log is:
`
Got exception outside of a @test
Gurobi Error 10001:
Stacktrace:
[1] _check_ret
@ ~/.julia/packages/Gurobi/6Pbhu/src/MOI_wrapper/MOI_wrapper.jl:400 [inlined]
[2] Gurobi.Env(; output_flag::Int64, memory_limit::Nothing, started::Bool)
@ Gurobi ~/.julia/packages/Gurobi/6Pbhu/src/MOI_wrapper/MOI_wrapper.jl:110
[3] Env
@ ~/.julia/packages/Gurobi/6Pbhu/src/MOI_wrapper/MOI_wrapper.jl:107 [inlined]
[4] Gurobi.Optimizer(env::Nothing; enable_interrupts::Bool)
@ Gurobi ~/.julia/packages/Gurobi/6Pbhu/src/MOI_wrapper/MOI_wrapper.jl:331
[5] Optimizer (repeats 2 times)
@ ~/.julia/packages/Gurobi/6Pbhu/src/MOI_wrapper/MOI_wrapper.jl:329 [inlined]
[6] _instantiate_and_check(optimizer_constructor::Any)
@ MathOptInterface ~/.julia/packages/MathOptInterface/wx5Ea/src/instantiate.jl:94
[7] instantiate(optimizer_constructor::Any; with_bridge_type::Type{Float64}, with_cache_type::Nothing)
@ MathOptInterface ~/.julia/packages/MathOptInterface/wx5Ea/src/instantiate.jl:175
[8] set_optimizer(model::JuMP.Model, optimizer_constructor::Any; add_bridges::Bool)
@ JuMP ~/.julia/packages/JuMP/9CBpS/src/optimizer_interface.jl:337
[9] JuMP.Model(optimizer_factory::Any; add_bridges::Bool)
@ JuMP ~/.julia/packages/JuMP/9CBpS/src/JuMP.jl:166
[10] Model
@ ~/.julia/packages/JuMP/9CBpS/src/JuMP.jl:165 [inlined]

`

[odow]

Your issue is that you care creating many Gurobi models in the global scope. you're right that these won't be freed by Julia's garbage collector and so you'll run out of memory.

So you have a couple of options.

The first is to move the loop body into a function, making sure that model doesn't leave the function. Then the memory associated with it will be freed after the function has been called:

function inner_loop(i)
    model = Model(Gurobi.Optimizer)
    set_optimizer_attribute(model, "Threads", option.thread)
    set_optimizer_attribute(model, "OutputFlag", option.log_level)
    set_optimizer_attribute(model, "TimeLimit", 10) # note Gurobi only supports wall-clock time
    set_silent(model)
    # variables
    @variables(model, begin 
        0 <= x <= 1 # 
        0 <= y <= 0.2 # 
    end) 
    # constraints
    @constraints(model, begin 
        x + y <=1
    end) 
    # objective
    @objective(model, Min, 0)
    optimize!(model)
end

for i in 1:10000
    inner_loop(i)
end

Alternatively, you could create a single model and modify it in each loop:

model = Model(Gurobi.Optimizer)
set_optimizer_attribute(model, "Threads", option.thread)
set_optimizer_attribute(model, "OutputFlag", option.log_level)
set_optimizer_attribute(model, "TimeLimit", 10) # note Gurobi only supports wall-clock time
set_silent(model)
@variables(model, begin 
    0 <= x <= 1 # 
    0 <= y <= 0.2 # 
end) 
@constraints(model, begin 
    con, x + y <=1
end) 
# objective
@objective(model, Min, 0)
for i in 1:10000
    set_normalized_rhs(con, i)
    optimize!(model)
end

[lidingxu]

Hi,
Both methods work fine. Thank you.