Reduce memory consumption of numpy function
$begingroup$
I have a Numpy function that takes the values of an existing array X and a size (int) k equivalent to the columns of the array. This function does some calculations in a new array, to finally output the new array after the calculations are done.
def ff(x, k):
newarr = np.zeros((x.shape[0], 1), dtype=np.uint32)
newarr[:, 0] = x[:, 0] * 4
for i in range(1, k - 1):
newarr[:, 0] += x[:, i]
newarr[:, 0] = newarr[:, 0] * 4
newarr[:, 0] += x[:, -1]
return newarr
newarr it's the array where the calculated data is stored, it's initialized with np.zeros and a dtype, and it has a shape of (y,1) where y is the length of the x array.
The first step is to store in the (only) column of the newarr the multiplication of the first column of the x by 4.
The iteration occurs over the columns 1 to k-1 of the original array (x).
Inside the iteration, the next step is to sum the values of the first column of x to the column of newarr.
Next, to multiply the column of newarr by 4
To finally, after the iteration is complete, sum the last column of the original array x to the column of newarr.
I'm looking for a way (if there's any) to avoid the creation of the newarr, and do the calculations in the original array. This', to reduce the memory usage of this function, mainly because the x tends to be very big in execution.
The x array looks like this.
array([[3, 3, 0, 1],
[3, 0, 1, 0],
[0, 1, 0, 2],
[1, 0, 2, 2],
[0, 2, 2, 3],
[2, 2, 3, 1],
[2, 3, 1, 3],
.
.
.
[2, 2, 0, 1]], dtype=uint8)
In this case, the k value would be 4
Any additional improvement in the code is well received!
Thanks for your time.
python-3.x numpy
asked 5 hours ago
KakoKako
8217
$endgroup$
add a comment |
$begingroup$
I have a Numpy function that takes the values of an existing array X and a size (int) k equivalent to the columns of the array. This function does some calculations in a new array, to finally output the new array after the calculations are done.
def ff(x, k):
newarr = np.zeros((x.shape[0], 1), dtype=np.uint32)
newarr[:, 0] = x[:, 0] * 4
for i in range(1, k - 1):
newarr[:, 0] += x[:, i]
newarr[:, 0] = newarr[:, 0] * 4
newarr[:, 0] += x[:, -1]
return newarr
newarr it's the array where the calculated data is stored, it's initialized with np.zeros and a dtype, and it has a shape of (y,1) where y is the length of the x array.
The first step is to store in the (only) column of the newarr the multiplication of the first column of the x by 4.
The iteration occurs over the columns 1 to k-1 of the original array (x).
Inside the iteration, the next step is to sum the values of the first column of x to the column of newarr.
Next, to multiply the column of newarr by 4
To finally, after the iteration is complete, sum the last column of the original array x to the column of newarr.
I'm looking for a way (if there's any) to avoid the creation of the newarr, and do the calculations in the original array. This', to reduce the memory usage of this function, mainly because the x tends to be very big in execution.
The x array looks like this.
array([[3, 3, 0, 1],
[3, 0, 1, 0],
[0, 1, 0, 2],
[1, 0, 2, 2],
[0, 2, 2, 3],
[2, 2, 3, 1],
[2, 3, 1, 3],
.
.
.
[2, 2, 0, 1]], dtype=uint8)
In this case, the k value would be 4
Any additional improvement in the code is well received!
Thanks for your time.
python-3.x numpy
asked 5 hours ago
KakoKako
8217
$endgroup$
add a comment |
$begingroup$
I have a Numpy function that takes the values of an existing array X and a size (int) k equivalent to the columns of the array. This function does some calculations in a new array, to finally output the new array after the calculations are done.
def ff(x, k):
newarr = np.zeros((x.shape[0], 1), dtype=np.uint32)
newarr[:, 0] = x[:, 0] * 4
for i in range(1, k - 1):
newarr[:, 0] += x[:, i]
newarr[:, 0] = newarr[:, 0] * 4
newarr[:, 0] += x[:, -1]
return newarr
newarr it's the array where the calculated data is stored, it's initialized with np.zeros and a dtype, and it has a shape of (y,1) where y is the length of the x array.
The first step is to store in the (only) column of the newarr the multiplication of the first column of the x by 4.
The iteration occurs over the columns 1 to k-1 of the original array (x).
Inside the iteration, the next step is to sum the values of the first column of x to the column of newarr.
Next, to multiply the column of newarr by 4
To finally, after the iteration is complete, sum the last column of the original array x to the column of newarr.
I'm looking for a way (if there's any) to avoid the creation of the newarr, and do the calculations in the original array. This', to reduce the memory usage of this function, mainly because the x tends to be very big in execution.
The x array looks like this.
array([[3, 3, 0, 1],
[3, 0, 1, 0],
[0, 1, 0, 2],
[1, 0, 2, 2],
[0, 2, 2, 3],
[2, 2, 3, 1],
[2, 3, 1, 3],
.
.
.
[2, 2, 0, 1]], dtype=uint8)
In this case, the k value would be 4
Any additional improvement in the code is well received!
Thanks for your time.
python-3.x numpy
asked 5 hours ago
KakoKako
8217
$endgroup$
I have a Numpy function that takes the values of an existing array X and a size (int) k equivalent to the columns of the array. This function does some calculations in a new array, to finally output the new array after the calculations are done.
def ff(x, k):
newarr = np.zeros((x.shape[0], 1), dtype=np.uint32)
newarr[:, 0] = x[:, 0] * 4
for i in range(1, k - 1):
newarr[:, 0] += x[:, i]
newarr[:, 0] = newarr[:, 0] * 4
newarr[:, 0] += x[:, -1]
return newarr
newarr it's the array where the calculated data is stored, it's initialized with np.zeros and a dtype, and it has a shape of (y,1) where y is the length of the x array.
The first step is to store in the (only) column of the newarr the multiplication of the first column of the x by 4.
The iteration occurs over the columns 1 to k-1 of the original array (x).
Inside the iteration, the next step is to sum the values of the first column of x to the column of newarr.
Next, to multiply the column of newarr by 4
To finally, after the iteration is complete, sum the last column of the original array x to the column of newarr.
I'm looking for a way (if there's any) to avoid the creation of the newarr, and do the calculations in the original array. This', to reduce the memory usage of this function, mainly because the x tends to be very big in execution.
The x array looks like this.
array([[3, 3, 0, 1],
[3, 0, 1, 0],
[0, 1, 0, 2],
[1, 0, 2, 2],
[0, 2, 2, 3],
[2, 2, 3, 1],
[2, 3, 1, 3],
.
.
.
[2, 2, 0, 1]], dtype=uint8)
In this case, the k value would be 4
Any additional improvement in the code is well received!
Thanks for your time.
python-3.x numpy
python-3.x numpy
asked 5 hours ago
KakoKako
8217
asked 5 hours ago
KakoKako
8217
asked 5 hours ago
KakoKako
8217
asked 5 hours ago
KakoKako
8217
asked 5 hours ago
KakoKako
8217
8217
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["\$", "\$"]]);
});
});
}, "mathjax-editing");
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "196"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f215783%2freduce-memory-consumption-of-numpy-function%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Code Review Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fcodereview.stackexchange.com%2fquestions%2f215783%2freduce-memory-consumption-of-numpy-function%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
